I'm working on a client library for a popular API. Currently, all of my unit tests of said client are making actual API calls against a test account.
Here's an example:
def test_get_foo_settings(self):
client = MyCustomClient(token, account)
results = client.get_foo_settings()
assert_is(type(results), list)
I'd like to stop making actual API calls against my test account.
How should I tackle this? Should I be using Mock to mock the calls to the client and response?
Also, I'm confused on the philosophy of what to test with this client library. I'm not interested in testing the actual API, but when there are different factors involved like the method being invoked, the permutations of possible return results, etc - I'm not sure what I should test and/or when it is safe to make assumptions (such as a mocked response).
Any direction and/or samples of how to use Mock in my type of scenario would be appreciated.
I would personally do it by first creating a single interface or function call which your library uses to actually contact the service, then write a custom mock for that during tests.
For example, if the service uses HTTP and you're using Requests to contact the service:
class MyClient(…):
def do_stuff(self):
result = requests.get(self.service_url + "/stuff")
return result.json()
I would first write a small wrapper around requests:
class MyClient(…):
def _do_get(self, suffix):
return requests.get(self.service_url + "/" + suffix).json()
def do_stuff(self):
return self._do_get("stuff")
Then, for tests, I would mock out the relevant functions:
class MyClientWithMocks(MyClient):
def _do_get(self, suffix):
self.request_log.append(suffix)
return self.send_result
And use it in tests like this:
def test_stuff(self):
client = MyClientWithMocks(send_result="bar")
assert_equal(client.do_stuff(), "bar")
assert_contains(client.request_log, "stuff")
Additionally, it would likely be advantageous to write your tests so that you can run them both against your mock and against the real service, so that if things start failing, you can quickly figure out who's fault it is.
I'm using HTTmock and I'm pretty happy with it : https://github.com/patrys/httmock
Related
In the following method the correct behaviour would to to save the news model before passing its id to the asynchronous Task queue for further processing.
Otherwise the task queue has the older version.
As you can see I have made a mistake and am saving the model after sending its id to the task queue.
def save_scraped_body_into_model(url_string):
news = ndb.Key(urlsafe=url_string).get()
...
news.body = result
news.stage = 1
taskqueue.Task(url='/api/v1.0/worker/bbc-stage-2', headers=header,
payload=json.dumps({'news_url_string': news.key.urlsafe()})).add(queue_name='newstasks')
news.put()
How do I possibly test for that?
My test below passes as long as the model was saved. But thats wrong, the order matters, and this test doesn't capture it !!! Is there a way to achieve this with mock?
def test_news_instance_saved_before_next_stage(self, get_head):
BBCSpider.save_scraped_body_into_model(self.news.key.urlsafe())
context = ndb.get_context()
context.clear_cache()
news = ndb.Key(urlsafe=self.news.key.urlsafe()).get()
self.assertEqual(news.stage, 1)
I'm doing these kind of tests by using patch from unittest.mock framework.
When I need to do test like this the first thing I do is looking for a point where I can put a hook. Then I patch the hook and use side_effect callback to do the test.
In your case the hook will be BBCSpider.taskqueue.Task and its side_effect something like this:
lambda *args,**kwargs: self.assertEqual(1, ndb.Key(urlsafe=self.news.key.urlsafe()).get().stage)
So your test become:
#patch('BBCSpider.taskqueue.Task', autospec=True)
def test_news_instance_saved_before_next_stage(self, get_head, mock_task):
def check_if_model_saved(*args,**kwargs):
news = ndb.Key(urlsafe=self.news.key.urlsafe()).get()
self.assertEqual(news.stage,1)
mock_task.side_effect = check_if_model_saved
BBCSpider.save_scraped_body_into_model(self.news.key.urlsafe())
self.assertTrue(mock_task.called)
Note autospec=True is not mandatory but I like to use it every time I do a patch to avoid silly errors.
I'm apologize if the code contains some mistake (I cannot test it without considerable effort) but I hope the idea is clear.
I have the following routing url rule defined and would like to test it.
app.add_url_rule('/api/v1.0/worker/bbc-stage-0', 'stage0', view_func=BBCStage0TaskView.as_view('bbc_stage0_taskview'))
The following tests if the path is correct:
def test_url_to_view_stage0_exists(self):
self.assertEqual(api.app.url_map._rules_by_endpoint['stage0'][0].rule, '/api/v1.0/worker/bbc-stage-0')
I haven't found a way to test if view_func is pointing to the right class. Is there a way to test that?
Werkzeug's Map maps paths to endpoints. The Flask app maps these endpoints to view functions in app.view_functions, which is used during app.dispatch_request. So to check what view has been connected to an endpoint, simply get it from that map. Since you're using a class based View, the real view function will be different every instantiation, so you instead test that the view_class is the same.
self.assertEqual(api.app.view_functions['stage0'].view_class, BBCStage0Task)
This is sort of a meaningless test, as you're basically testing Flask internals, which are already tested by Flask. Your own tests would be much more useful by simply using the test client to see if a request to a url returns what you expect.
with api.app.test_client() as client:
rv = client.get('/api/v1.0/worker/bbc-stage-0')
# assert something about the response, such as 'expected string' in rv.data
I am currently trying to write a little web-application using python, flask and sqlite and I'm not sure about how to handle the database-connections.
Basically I've been following the "official" Tutorial (and http://flask.pocoo.org/docs/patterns/sqlite3/#sqlite3 ), leaving me with code like this in my main Flask/App module (drastically shortened):
#vs_app.before_request
def before_request():
g.db = sqlite3.connect("somedb.db")
def execute_db(command):
return g.db.cursor().execute(command).fetchall()
#app.route("/givemeallusers")
def foo():
return execute_db("SELECT * FROM users")
So this creates a DB-Connection for each request that can be used by my application to execute sql-commands and it works fine for smaller applications.
But for a larger project I'd like to put my database-code (with some utility methods) and app-code in different modules, so I could (for example) write:
from my_database_module import user_auth #
#app.route("/login")
def foo():
if user_auth(request.form["name"], request.form["pw"]):
pass
But for that I would need access to g (or g.db) from my database-class and the only way I could get that to work was by "manually" handing over the db-connection to each method, leaving me with code like this:
#app.route("/login")
def foo():
if user_auth(g.db, request.form["name"], request.form["pw"]):
pass
and in my database-module
def user_auth(database, name, pw):
pass
I don't think thats the best approach, but other ideas I had (like importing the g object into my database-class) don't work. I also don't know whether my approach is safe with regards to concurrent db-access, so any help with this would be appreciated.
tl;dr How to split Flask-App and Database/Database-Utility the right way?
Just in case someone stumbles upon this question:
The correct answer is to not bother with the approaches described in the Question, and instead just start working with Flask-SQLAlchemy.
I've written a program in Python, which works with two distinct API to get the data from two different services (CKAN and MediaWiki).
In particular, there is a class Resource, which requests the data from the above mentioned services and process it.
At some point I've come to conclusion, that there is a need for tests for my app.
And the problem is that all examples I've found on web and in books do not deal with such cases.
For example, inside Resource class I've got a method:
def load_from_ckan(self):
"""
Get the resource
specified by self.id
from config.ckan_api_url
"""
data = json.dumps({'id': self.id})
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
url = config.ckan_api_url + '/action/resource_show'
r = requests.post(url, timeout=config.ckan_request_timeout, data=data, headers=headers)
assert r.ok, r
resource = json.loads(r.content)
resource = resource["result"]
for key in resource:
setattr(self, key, resource[key])
The load_from_ckan method get the data about resource from the CKAN API and assign it to the object. It is simple, but...
My question is: how to test the methods like this? OR What should I test here?
I thought about the possibility to pickle (save) results to HDD. Then I could load it in the test and compare with the object initialized with load_from_ckan(). But CKAN is community-driven platform and such behavior of such tests will be unpredictable.
If there exist any books on philosophy of automated tests (like what to test, what not to test, how to make tests meaningful etc.), please, give me a link to it.
With any testing, the key question really is - what could go wrong?
In your case, it looks like the three risks are:
The web API in question could stop working. But you check for this already, with assert r.ok.
You, or someone else, could make a mistaken change to the code in future (e.g. mistyping a variable) which breaks it.
The API could change, so that it no longer returns the fields or the format you need.
It feels like you could write a fairly simple test for the latter two, depending on what data from this API you actually rely on: for example, if you're expecting the JSON to have a field called "temperature" which is a floating-point Celsius number, you could write a test which calls your function, then checks that self.temperature is an instance of 'float' and is within a sensible range of values (-30 to 50?). That should leave you confident that both the API and your function are working as designed.
Typically if you want to test against some external service like this you will need to use a mock/dummy object to fake the api of the external service. This must be configurable at run-time either via the method's arguments or the class's constructor or another type of indirection. Another more complex option would be to monkey patch globals during testing, like "import requests; request.post = fake_post", but that can create more problems.
So for example your method could take an argument like so:
def load_from_ckan(self, post=requests.post):
# ...
r = post(url, timeout=config.ckan_request_timeout, data=data,
headers=headers)
# ...
Then during testing your would write your own post function that returned json results you'd see coming back from ckan. For example:
def mock_post(url, timeout=30, data='', headers=None):
# ... probably check input arguments
class DummyResponse:
pass
r = DummyResponse()
r.ok = True
r.content = json.dumps({'result': {'attr1': 1, 'attr2': 2}})
return r
Constructing the result in your test gives you a lot more flexibility than pickling results and returning them because you can fabricate error conditions or focus in on specific formats your code might not expect but you know could exist.
Overall you can see how complicated this could become so I would only start adding this sort of testing if you are experiencing repeated errors or other difficulties. This will just more code you have to maintain.
At this point, you can test that the response from CKAN is properly parsed. So you can pull the JSON from CKAN and ensure that it's returning data with the attributes you're interested in.
In general I want to disable as little code as possible, and I want it to be explicit: I don't want the code being tested to decide whether it's a test or not, I want the test to tell that code "hey, BTW, I'm running a unit test, can you please not make your call to solr, instead can you please stick what you would send to solr in this spot so I can check it". I have my ideas but I don't like any of them, I am hoping that there's a good pythonic way to do this.
You can use Mock objects to intercept the method calls that you do not want to execute.
E.g. You have some class A, where you don't want method no() to be called during a test.
class A:
def do(self):
print('do')
def no(self):
print('no')
A mock object could inherit from A and override no() to do nothing.
class MockA(A):
def no(self):
pass
You would then create MockA objects instead of As in your test code. Another way to do mocking would be to have A and MockA implement a common interface say InterfaceA.
There are tons of mocking frameworks available. See StackOverflow: Python mocking frameworks.
In particular see: Google's Python mocking framework.
Use Michael Foord's Mock
in your unit test do this:
from mock import Mock
class Person(object):
def __init__(self, name):
super(Person, self).__init__()
self.name = name
def say(self, str):
print "%s says \"%s\"" % (self.name, str)
...
#In your unit test....
#create the class as normal
person = Person("Bob")
#now mock all of person's methods/attributes
person = Mock(spec=person)
#talkto is some function you are testing
talkTo(person)
#make sure the Person class's say method was called
self.assertTrue(person.say.called, "Person wasn't asked to talk")
#make sure the person said "Hello"
args = ("Hello")
keywargs = {}
self.assertEquals(person.say.call_args, (args, keywargs), "Person did not say hello")
The big problem that I was having was with the mechanics of the dependency injection. I have now figured that part out.
I need to import the module in the exact same way in both places to successfully inject the new code. For example, if I have the following code that I want to disable:
from foo_service.foo import solr
solr.add(spam)
I can't seem to do this in the in my test runner:
from foo import solr
solr = mock_object
The python interpreter must be treating the modules foo_service.foo and foo as different entries. I changed from foo import solr to the more explicit from foo_service.foo import solr and my mock object was successfully injected.
Typically when something like this arises you use Monkey Patching (also called Duck Punching) to achieve the desired results. Check out this link to learn more about Monkey Patching.
In this case, for example, you would overwrite solr to just print the output you are looking for.
You have two ways to do this is no ,or minimal in the case of DI, modifications to your source code
Dependency injection
Monkey patching
The cleanest way is using dependency injection, but I don't really like extensive monkeypatching, and there are some things that are non-possible/difficult to do that dependency injection makes easy.
I know it's the typical use case for mock objects, but that's also an old argument... are Mock objects necessary at all or are they evil ?
I'm on the side of those who believe mocks are evil and would try to avoid changing tested code at all. I even believe such need to modify tested code is a code smell...
If you wish to change or intercept an internal function call for testing purpose you could also make this function an explicit external dependency set at instanciation time that would be provided both by your production code and test code. If you do that the problem disappear and you end up with a cleaner interface.
Note that doing that there is not need to change the tested code at all neither internally nor by the test being performed.