I'm using the requests module in my code (obviously to do requests), and my code is quickly getting out of hand because of redundant parameters that I need to include for each request:
def one(url, data, headers, cert):
...
return requests.post(url, json=data, headers=headers, verify=cert)
def two(otherurl, otherheaders, cert):
...
response = requests.get(otherurl, headers=otherheaders, verify=cert).json()
Is there a way to tell every request to use verify=cert without having to include it within every request statement? I'm thinking session() should be able to do this, although I have no idea how to use it. I'm just trying to minimize the repeating of things that maybe could be set globally within my script. Maybe this is not possible or how it actually works? thanks in advance.
You can use functools.partial to override these functions with verify=cert passed as an argument by default:
from functools import partial
requests.post = partial(requests.post, verify=cert)
requests.get = partial(requests.get, verify=cert)
Or if you look at the source code of requests, you'll find that both of these two functions are simply wrappers to the requests.request function, which in turn is a wrapper to the requests.Session.request method. You can therefore override requests.Session.request instead to have all the HTTP methods overridden with one statement. Since it's a method and not an unbound function, however, you have to use functools.partialmethod instead:
from functools import partialmethod
requests.Session.request = partialmethod(requests.Session.request, verify=cert)
Related
My function is below:
def do(self, text, disable=['ner'], all_tokens=False, char_match=True, channels=use_channels)
Now I am using a Flask http call to make post requests to the function, and the function parameters are passed through the http call. The results of parameters is in a dict:
parameters = {'disable':['ner', 'lst'], 'all_tokens':True, 'char_match':False}
My question is, how to apply the parameters in the dict to the function 'do'?
If I'm understanding you correctly, all you need to do is unpack the parameters object in the 'do' function. E.g.
do(**parameters)
If you're talking about how to pull the parameters from the URL-
You'll need to get them one at a time IIRC, but as follows:
from flask import request
disable = request.args.get('disable')
all_tokens = request.args.get('all_tokens')
...
do(..., disable=disable, all_tokens=all_tokens)
I'm messing around with unittest.mock and got some problems with it.
I have an object client with method get_messages() which returns response with attributes data and has_more. I want to mock it to return fixed data and has_more in first call and another fixed data and has_more in second call.
In first call I want to receive object response with attributes:
data=['msg1', 'msg2']
has_more=True
In second call I want to receive object response with attributes:
data=['msg3', 'msg4']
I've been trying doing it this way, but I'm kind of confused, no idea if this is the way.
#patch('Client')
def test_client_returns_correct_messages(self, MockClient):
MockWebClient.get_messages.side_effects = [
Mock(name='response',
data={'messages': received_messages,
'has_more': True}),
Mock(name='response',
data={'messages': received_messages,
'has_more': False})]
messages = client.get_messages()
Okay, I found the answer... Generally my code was fine, but I made a typo: side_effects instead of side_effect - NOTE THE s. It should be side_effect. Mock accepts everything, so it didn't rise an error. Will definitely use specs next time :D I still don't know if this is the correct way to do this, but it works.
this is the working code:
#patch('Client')
def test_client_returns_correct_messages(self, MockClient):
MockWebClient.get_messages.side_effect = [
Mock(name='response',
data={'messages': received_messages,
'has_more': True}),
Mock(name='response',
data={'messages': received_messages,
'has_more': False})]
messages = client.get_messages()
Accordig to docs
If you pass in an iterable, it is used to retrieve an iterator which must yield a value on every call. This value can either be an exception instance to be raised, or a value to be returned from the call to the mock (DEFAULT handling is identical to the function case).
Here is an example of a call, twice, from a same method but with two different answers.
import os
from unittest.mock import patch
#patch('os.path.curdir', side_effect=[True, False])
def test_side_effect(mock_curdir):
print(os.path.curdir())
print(os.path.curdir())
>>> test_side_effect()
True
False
I'm creating a test suite for my module that uses the requests library quite a bit. However, I'm trying to mock several different return values for a specific request, and I'm having trouble doing so. Here is my code snippet that doesn't work:
class MyTests(unittest.TestCase):
#patch('mypackage.mymodule.requests.post')
def test_change_nested_dict_function(self, mock_post):
mock_post.return_value.status_code = 200
mock_post.return_value.json = nested_dictionary
modified_dict = mymodule.change_nested_dict()
self.assertEqual(modified_dict['key1']['key2'][0]['key3'], 'replaced_value')
The function I am attempting to mock:
import requests
def change_nested_dict():
uri = 'http://this_is_the_endpoint/I/am/hitting'
payload = {'param1': 'foo', 'param2': 'bar'}
r = requests.post(uri, params=payload)
# This function checks to make sure the response is giving the
# correct status code, hence why I need to mock the status code above
raise_error_if_bad_status_code(r)
dict_to_be_changed = r.json()
def _internal_fxn_to_change_nested_value(dict):
''' This goes through the dict and finds the correct key to change the value.
This is the actual function I am trying to test above'''
return changed_dict
modified_dict = _internal_fxn_to_change_nested_value(dict_to_be_changed)
return modified_dict
I know a simple way of doing this would be to not have a nested function, but I am only showing you part of the entire function's code. Trust me, the nested function is necessary and I really do not want to change that part of it.
My issue is, I don't understand how to mock requests.post and then set the return value for both the status code and the internal json decoder. I also can't seem to find a way around this issue since I can't seem to patch the internal function either, which also would solve this problem. Does anyone have any suggestions/ideas? Thanks a bunch.
I bumped here and although I agree that possibly using special purpose libraries is a better solution, I ended up doing the following
from mock import patch, Mock
#patch('requests.post')
def test_something_awesome(mocked_post):
mocked_post.return_value = Mock(status_code=201, json=lambda : {"data": {"id": "test"}})
This worked for me for both getting the status_code and the json() at the receiver end while doing the unit-test.
Wrote it here thinking that someone may find it helpful.
When you mock a class each child method is set up as a new MagicMock that in turn needs to be configured. So in this case you need to set the return_value for mock_post to bring the child attribute into being, and one to actually return something, i.e:
mock_post.return_value.status_code.return_value = 200
mock_post.return_value.json.return_value = nested_dictionary
You can see this by looking at the type of everything:
print(type(mock_post))
print(type(mock_post.json))
In both cases the type is <class 'unittest.mock.MagicMock'>
Probably it is better for you to look at some specialized libraries for requests testing:
responses
requests-mock
requests-testing
They provide clean way to mock responses in unittests.
An alternate approach is to just create an actual Response object and then do a configure_mock() on the original mock.
from requests import Response
class MyTests(unittest.TestCase):
#patch('mypackage.mymodule.requests.post')
def test_change_nested_dict_function(self, mock_post):
resp = Response()
resp.status_code = 200
resp.json = nested_dictionary
mock_post.configure_mock(return_value=resp)
...
I'm trying to implement a batch request method in pyramid. I see in the docs that it's done with
subrequest = Request.blank('/relative/url')
request.invoke_subrequest(subrequest)
I'm just wondering how do you pass along the headers and cookies? Is it already done for you or is it
request.invoke_subrequest(subrequest, cookies=request.cookies, headers=request.headers)
What about parameters for different methods? The docs only have a POST keyword arg.
I feel like the docs are a little vague, or I can't find the correct docs on how to do this. Thanks
I'm just wondering how do you pass along the headers and cookies?
From http://docs.pylonsproject.org/projects/pyramid/en/1.5-branch/narr/subrequest.html#subrequest-chapter :
The pyramid.request.Request.invoke_subrequest() API accepts two
arguments: a positional argument request that must be provided, and
use_tweens keyword argument that is optional; it defaults to False.
This tells us that the constructor only wants a Request object, and optionally a value for use_tweens, so no, this
request.invoke_subrequest(subrequest, cookies=request.cookies, headers=request.headers)
will not work.
Then, from http://docs.pylonsproject.org/projects/pyramid/en/1.5-branch/narr/subrequest.html
It's a poor idea to use the original request object as an argument to
invoke_subrequest(). You should construct a new request instead as
demonstrated in the above example, using
pyramid.request.Request.blank(). Once you've constructed a request
object, you'll need to massage it to match the view callable you'd
like to be executed during the subrequest. This can be done by
adjusting the subrequest's URL, its headers, its request method, and
other attributes. The documentation for pyramid.request.Request
exposes the methods you should call and attributes you should set on
the request you create to massage it into something that will actually
match the view you'd like to call via a subrequest.
So basically, you need to configure your request before you pass it to invoke_subrequest().
Luckily there is an entire page that documents the Request class. There we can find a whole lot options to configure it, etc.
What about parameters for different methods? The docs only have a POST keyword arg.
Also on the Request class documentation page, there is this
method
Gets and sets the REQUEST_METHOD key in the environment.
And on http://docs.pylonsproject.org/projects/pyramid/en/1.3-branch/narr/viewconfig.html I found
request_method This value can be a string (typically "GET", "POST",
"PUT", "DELETE", or "HEAD") representing an HTTP REQUEST_METHOD
I must agree with you that the documentation is a little vague here and there, but I assume you can use it like this
request.method = 'POST'
# Or
request.method = 'GET'
# Etc.
Summary
You'll want to do it like this
subrequest = Request.blank('/relative/url')
# Configure the subrequest object
# set headers and other options you need.
request.invoke_subrequest(subrequest)
Note
I am aware this is not a 100% complete answer with some code that you can copy paste and it'll just work (especially regarding configuring the request object), but I think this answer contains some information that, at the very least, will get you on the right track and I hope it will be of some help to you.
The following code worked for me. It copies all (headers, cookies, query string, post parameters, etc.):
def subrequest(request, path):
subreq = request.copy()
subreq.path_info = path
response = request.invoke_subrequest(subreq)
return response
Somewhat late, but based on the above two answers here is how I did this. I didn't quite like the above answer to just copy everything. Looking at the documentation of the blank() method there is a kw argument and it says
All necessary keys will be added to the environ, but the values you pass in will take precedence. If you pass in base_url then wsgi.url_scheme, HTTP_HOST, and SCRIPT_NAME will be filled in from that value.
Assuming that the view's request contains the right header information and cookies that you need for your subrequest, you can use the following code:
#view_config( ... )
def something(request):
...
kwargs = { "cookies": request.cookies,
"host" : request.host,
}
req = Request.blank("/relative/url", **kwargs)
resp = request.invoke_subrequest(req)
Other header information (e.g. accept, accept_encoding, etc.) are properties of pyramid.request objects, and can be added to the kwargs dictionary like shown in the code snippet above.
The object returned by invoke_subrequest() is a Response object documented here.
I've written a program in Python, which works with two distinct API to get the data from two different services (CKAN and MediaWiki).
In particular, there is a class Resource, which requests the data from the above mentioned services and process it.
At some point I've come to conclusion, that there is a need for tests for my app.
And the problem is that all examples I've found on web and in books do not deal with such cases.
For example, inside Resource class I've got a method:
def load_from_ckan(self):
"""
Get the resource
specified by self.id
from config.ckan_api_url
"""
data = json.dumps({'id': self.id})
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
url = config.ckan_api_url + '/action/resource_show'
r = requests.post(url, timeout=config.ckan_request_timeout, data=data, headers=headers)
assert r.ok, r
resource = json.loads(r.content)
resource = resource["result"]
for key in resource:
setattr(self, key, resource[key])
The load_from_ckan method get the data about resource from the CKAN API and assign it to the object. It is simple, but...
My question is: how to test the methods like this? OR What should I test here?
I thought about the possibility to pickle (save) results to HDD. Then I could load it in the test and compare with the object initialized with load_from_ckan(). But CKAN is community-driven platform and such behavior of such tests will be unpredictable.
If there exist any books on philosophy of automated tests (like what to test, what not to test, how to make tests meaningful etc.), please, give me a link to it.
With any testing, the key question really is - what could go wrong?
In your case, it looks like the three risks are:
The web API in question could stop working. But you check for this already, with assert r.ok.
You, or someone else, could make a mistaken change to the code in future (e.g. mistyping a variable) which breaks it.
The API could change, so that it no longer returns the fields or the format you need.
It feels like you could write a fairly simple test for the latter two, depending on what data from this API you actually rely on: for example, if you're expecting the JSON to have a field called "temperature" which is a floating-point Celsius number, you could write a test which calls your function, then checks that self.temperature is an instance of 'float' and is within a sensible range of values (-30 to 50?). That should leave you confident that both the API and your function are working as designed.
Typically if you want to test against some external service like this you will need to use a mock/dummy object to fake the api of the external service. This must be configurable at run-time either via the method's arguments or the class's constructor or another type of indirection. Another more complex option would be to monkey patch globals during testing, like "import requests; request.post = fake_post", but that can create more problems.
So for example your method could take an argument like so:
def load_from_ckan(self, post=requests.post):
# ...
r = post(url, timeout=config.ckan_request_timeout, data=data,
headers=headers)
# ...
Then during testing your would write your own post function that returned json results you'd see coming back from ckan. For example:
def mock_post(url, timeout=30, data='', headers=None):
# ... probably check input arguments
class DummyResponse:
pass
r = DummyResponse()
r.ok = True
r.content = json.dumps({'result': {'attr1': 1, 'attr2': 2}})
return r
Constructing the result in your test gives you a lot more flexibility than pickling results and returning them because you can fabricate error conditions or focus in on specific formats your code might not expect but you know could exist.
Overall you can see how complicated this could become so I would only start adding this sort of testing if you are experiencing repeated errors or other difficulties. This will just more code you have to maintain.
At this point, you can test that the response from CKAN is properly parsed. So you can pull the JSON from CKAN and ensure that it's returning data with the attributes you're interested in.