Related to Python 2.7
How would one go about building a request through a variable number of kwargs when using requests.
I am using the requests module to directly interact with a REST API which requires a variable number of keyword arguments in order to be successful.
Rather than re-writing the same GET/POST request code, I would like to maintain it within a single api class. However handling the variable number of arguments seems to boil down to a series of if-else statements which isn't particularly readable.
For example:
def request(self):
try:
if self.data:
request = requests.post(url=self.url, headers=self.headers,
data=self.data, timeout=self.timeout, verify=False)
else:
request = requests.get(url=self.url, headers=self.headers,
timeout=self.timeout, verify=False)
...
...
Preferably the request properties are build over time and them passed through a single GET or POST request (granted, the above code would still be require but that is minor).
If you make the attributes default to the same values as arguments to requests.post (basically, None), than you can safely pass all of them as keyword arguments:
def request(self):
try:
request = requests.post(url=self.url, headers=self.headers,
data=self.data, timeout=self.timeout,
verify=False)
...
Related
I have created a an application in python-django, where it makes some api call to some 3rd party services and then return response, post that I am doing some processing on response data and generating some final document. Below is something I am doing:
def get_processed_data(url, payload, tenant, req_id, timeout=None):
query_kwargs = HTTPRequestUtils.__get_query_kwargs(timeout)
query_kwargs['json'] = payload
response = HTTPRequestUtils.__get_response(url, query_kwargs, requests.post)
.....
data=process_response(response)
return more_processings(data)
Abobe is one of the function , being called during actual execution of code. And response varies with url.
Now problem is I have am writing Unit Test , and i have to emulate/mock http call, so that for different url, i may return different mocked response, that will be further processed.
I went through several libraries like responses etc, but what I conclude from them is , i can test just api call and its returned response. But in actual I need to just emulate/mock http call for different and return response back, like we do in patch during mock, so that the response can be further go for processing.
Any library or method by which we can achieve this.
If you have an idea of order in which the API call would take place, you can make use of side-effect func of mock library, so what it does is it will give different response for each time the mock function is called
for eg:
mock_api.side_effect = [(resp1),(resp2)]
so when api() will be called for the 1st time => response will be resp1 and for the second time ==> response will be resp2
I think this will solve your problem
I'm learning Python requests through a book I purchased. From the book and from the websites I researched, it states this is the proper way to perform a GET request.
requests.get(url, params={key: value}, args)
In PyCharm it also shows the same thing.
However, when I use Postman to test and look at the sample code it shows:
requests.request("GET", params={key: value}, args
I'm not sure if I should use request.get or requests.request("GET",`
and why choose one over the other.
They are both correct and will work the same.
The best way to clear up this confusion is to look at the requests source code.
Here is the code for request.get (as of 2.25.1):
def get(url, params=None, **kwargs):
r"""Sends a GET request.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary, list of tuples or bytes to send
in the query string for the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
kwargs.setdefault('allow_redirects', True)
return request('get', url, params=params, **kwargs)
...which shows that requests.get just calls requests.request with a hardcoded 'get' for the 1st argument. All the other parameters (url, params, **kwargs) are all just passed through.
Basically, it is just a convenience method or a shorthand or a shortcut so you don't have to manually remember which string to pass for the method parameter. It's easier especially when using an IDE because your IDE's IntelliSense can help you select .get or .post or .delete, etc. but not the raw strings "GET", "POST", or "DELETE", etc.
The requests docs can also offer some clarity.
requests.request(method, url, **kwargs)
It says "Constructs and sends a Request.". So this one is for ANY type of request, and you need to pass in 2 required arguments: the method and the URL. All the possible kwargs are listed in the doc, including the params from your example.
requests.get(url, params=None, **kwargs)
It says "Sends a GET request.". So this one is more specific in that it is only for a GET request. It only has 1 required argument, the URL. No need to pass "GET". And then kwargs is "Optional arguments that request takes." which just points back to the main requests.request method.
I would say it's a matter of opinion and coding style which one to use. A use-case where it makes sense to prefer requests.request is when wrapping different API calls and providing a Python interface for them.
For example, I have these APIs:
GET /api/v1/user/[user-id]
PATCH /api/v1/user/[user-id]
DELETE /api/v1/user/[user-id]
When implementing get_user and update_user and delete_user, I could just call the .get, .post, .delete, etc. methods. But if calling these APIs required passing in many and/or complicated kwargs (headers, auth, timeout, etc.), then that would lead to a lot of duplicated code.
Instead, you can do it all in one method then use requests.request:
def get_user(user_id):
return call_api("GET", user_id)
def update_user(user_id, user):
return call_api("PATCH", user_id, user=user)
def delete_user(user_id):
return call_api("DELETE", user_id)
def call_api(method, user_id, user=None):
# Lots of preparation code to make a request
headers = {
"header1": "value1",
"header2": "value2",
# There can be lots of headers...
"headerN": "valueN",
}
timeout = (1, 30)
payload = user.json() if user else {}
url = f"/api/v1/user/{user_id}"
return requests.request(
method,
url,
headers=headers,
timeout=timeout,
json=payload,
)
There are probably other ways to refactor the code above to reduce duplication. That's just an example, where if you called .get, .patch, .delete directly, then you might end up repeating listing all those headers every time, setting up the URL, doing validations, etc.
I want to unit test my Python Azure function. I'm following the Microsoft documentation.
The documentation mocks the call to the function as follows
req = func.HttpRequest(
method='GET',
body=None,
url='/api/HttpTrigger',
params={'name': 'Test'})
I would like to do this but with the parameters passed as a JSON object so that I can follow the req_body = req.get_json() branch of the function code. I guessed I would be able to do this with a function call like
req = func.HttpRequest(
method='GET',
body=json.dumps({'name': 'Test'}),
url='/api/HttpTrigger',
params=None)
If I construct the call like this, req.get_json() fails with the error message AttributeError: 'str' object has no attribute 'decode'.
How do I construct the request with JSON input parameters? It should be trivial but I'm clearly missing something obvious.
If I construct my mock call as follows:
import json
req = func.HttpRequest(
method='POST',
body=json.dumps({'name': 'Test'}).encode('utf8'),
url='/api/HttpTrigger',
params=None)
Then I am able to make a successful call to req.get_json(). Thanks to #MrBeanBremen and #JoeyCai for pointing me in the correct direction i.e. don't call GET and make the message a byte string.
Any HTTP request message is allowed to contain a message body, and thus must parse messages with that in mind. Server semantics for GET, however, are restricted such that a body, if any, has no semantic meaning to the request. The requirements on parsing are separate from the requirements on method semantics.
For your http request, it is a GET method. You can send a request body with GET but it should not have any meaning.
So use the below code to construct a mock HTTP request with a json payload. For more details, you could refer to this article.
req = func.HttpRequest(
method='GET',
body=None,
url='/api/HttpTrigger',
params={'name': 'Test'})
Update:
For Post request, you could send json payload with body=json.dumps({'name': 'Test'}).encode('utf8') while body expects a byte string:
req = func.HttpRequest(
method='POST',
body=json.dumps({'name': 'Test'}).encode('utf8'),
url='/api/HttpTrigger',
params=None)
We have some custom module where we have redefined open, seek, read, tell functions to read only a part of file according to the arguments.
But, this logic overrides the default tell and python requests is trying to calculate the content-length which involves using tell(), which then redirects to our custom tell function and the logic is somewhere buggy and returns a wrong value. And I tried some changes, it throws error.
Found the following from models.py of requests:
def prepare_content_length(self, body):
if hasattr(body, 'seek') and hasattr(body, 'tell'):
body.seek(0, 2)
self.headers['Content-Length'] = builtin_str(body.tell())
body.seek(0, 0)
elif body is not None:
l = super_len(body)
if l:
self.headers['Content-Length'] = builtin_str(l)
elif (self.method not in ('GET', 'HEAD')) and (self.headers.get('Content-Length') is None):
self.headers['Content-Length'] = '0'
For now, I am not able to figure out where's the bug and stressed out to investigate more and fix it. And everything else work except content-length calculation by python requests.
So, I have created my own definition for finding content-length. And I have included the value in requests header. But, the request is still preparing the content-length and throwing error.
How can I restrict not preparing content-length and use the specified content-length?
Requests lets you modify a request before sending. See Prepared Requests.
For example:
from requests import Request, Session
s = Session()
req = Request('POST', url, data=data, headers=headers)
prepped = req.prepare()
# do something with prepped.headers
prepped.headers['Content-Length'] = your_custom_content_length_calculation()
resp = s.send(prepped, ...)
If your session has its own configuration (like cookie persistence or connection-pooling), then you should use s.prepare_request(req) instead of req.prepare().
The code called by my script is (this is code in api.py)
def post(url, data=None, **kwargs):
"""Sends a POST request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
"""
return request('post', url, data=data, **kwargs)
Now I need to POST a request which has no payload, as the info which I need to post is in url itself. I have tried following combinations but failed:
1) requests.post(url, auth, data=None)
Fails saying:
result = requests.post(api, auth, data=None)
TypeError: post() got multiple values for keyword argument 'data'
2) requests.post(api, auth, data=payload) where payload is empty json.
Please suggest..
Passed auth param is accepted by function as data one. Then you passed data again, as keyword argument.
result = requests.post(api, auth, data=None)
TypeError: post() got multiple values for keyword argument 'data'
Try this:
result = requests.post(api, auth=auth)
The issue is the incompatibility between the method signature and how you call it.
def post(url, data=None, **kwargs):
result = requests.post(api, auth, data=None)
First of all, based on the error, its save to assume requests is a library you've written (and not the Python requests module), and not a class because you'd get a completely different error otherwise.
Your post method has 2 arguments, url and data.
Your call has three arguments, which python has to unpack: the variables api and auth and a named argument of data=None.
Python assigns api to the url variable in the methods scope, and auth to the data variable. This leaves an extra named data variable in the methods scope, which now is attempting to be assigned again.
Hence the error:
post() got multiple values for keyword argument 'data'