The Python unittest doc defines a test case as:
"[...] the smallest unit of testing. It checks for a specific response to a
particular set of inputs."
However the first example contains a method with two assertions:
def test_shuffle(self):
...
self.assertEqual(self.seq, range(10))
...
self.assertRaises(TypeError, random.shuffle, (1,2,3))
This is clearly a contradiction, since the assertions each contain their own inputs and expected response.
Which approach is most appropriate?
Having two assertions is not a problem in and of itself. But the first example in the docs does test two separate things, and should be split into smaller pieces.
The question of how finely to slice your tests can become a philosophical one. Find a balance that feels right to you.
Related
How do I write a test to validate if a string manipulation was called or not? In this specific situation I'm trying to test that upper was called at least once. Since this is a python built in method it's a little different and I can't wrap my head around it.
# my function that returns an uppercase string example
def my_upper(str_to_upper: str) -> str:
return str(str_to_upper).upper()
# my test that should determine that .upper() was called
def test_my_upper():
# i assume I need some kind of mock here?
my_upper('a')
assert upper.call_count == 1
Update: I need to know if the core implementation of a very large product has changed. If I implement string manipulation and another dev comes in and changes how it works I want tests to immediately let me know so I can verify the implementation they added works or not.
Another update: here's what I've tried. It's complaining it can't find library 'str'.
from mock import patch
#patch("str.upper")
def test_my_upper(mock_upper):
my_upper('a')
assert mock_upper.call_count == 1
Answer for the main question: you can use pytest spy
Reference: https://pypi.org/project/pytest-mock/
def test_spy_method(mocker):
spy = mocker.spy(str, 'upper')
spy.assert_has_called_once(21) # more advanced method could be used such as the ones with count
Answer the Update: It depends entirely of the implemented tests, if you have a plenty of ones which tests each of one the behaviors from it you MIGHT get a change. But if the change do something that keeps the tests results maybe tests will continue to work. Nitpick, tests isn't for this kind of matter, but to check the good behavior of a piece of software Thus, if the code changes and regression tests continue work as expected within the threshold coverage that shouldn't be a problem.
Answer for the another Update: Indeed, str isn't an imported object, at least not that you are used to be. And for what I've understood from the question you want to know about calls from a given method from str, this use cases fits perfectly by spy. Another point, is that you don't need the create a wrapper for a method to get something from test, actually the code should "live" a part from tests.
How can I unit test variables/values that are inside a function?
This is a rather basic question about unit testing (I am using pytest) that is I want to make sure all my code behaves as expected.
There must be a proper way to do that, but I didn't find out. As of yet, I try to split up my code in as many functions as possible to get as many return values as possible that I can test. But I am not able to test inside those functions.
Here I can only test if the return value interp_function is working as expected, but in no way to test the rest of the code.
def multidim_interp_function(grid = None, max_error_in_mm=1,
max_error_in_deg=1):
def interp_function(array_of_pts):
fct_list = create_interp(grid)
max_error = [max_error_in_mm]*3
if len(grid) == 6:
max_error.extend([max_error_in_deg]*3)
return np.column_stack([error*fct_list[i](array_of_pts) for i,error in enumerate(max_error)])
return interp_function
You don't need to test the implementation of your function. If you wrote foo = 'bar' inside your function, then you don't need to test whether foo correctly has been assigned the value 'bar'; you can just expect that to work. With unit tests you want to be one step more abstract. You want to check whether your function multidim_interp_function returns the correct results given some known input. Treat the function like you'd treat other functions in Python: you wouldn't write a unit test to figure out how max() works internally, instead you'd write a test asserting that max(3, 4) returns the value 4.
Not only is it impractical to test the "internals" of a function, it's that the internals of a function may change. If you're writing your unit tests and you figure out you have some bug in your code, then you're going to change your code. Or you may later come back and refactor the internals to make them more efficient, or to deduplicate some code in your module, or whatever. You wouldn't want to rewrite your unit tests each time. So you shouldn't write your unit tests to be too specific. Your unit tests should test the public interface of the function (arguments and return values), so you can assure that this interface doesn't change (i.e. the function continues to behave the same) even if you move some code around. If through your unit test or otherwise you figure out that there’s some misbehavior inside the function, then you can step in with a debugger and confirm each single statement one by one to find where an issue is.
To get into the right mindset, try Test-Driven Development, in which you write your tests firsts, essentially deciding on how the function should behave, and then you implement the function internals, all the while being able to test that your implementation conforms to the expectation.
Basically, you want your function to behave the same and so, results should be predicted. You don't want to test internal state of your function, but rather the output.
def my_calculus(divise, multiply):
def inner(value):
return value * multiply / divide
return inner
calculus = my_calculus(2, 4)
#pytest.mark.parameterize("function,value,expected", [
(calculus, 2, 4),
(calculus, 5, 10)
])
def test_my_calculus(function, value, expected):
assert function(value) == excepted
I have the following code:
def task_completed(task):
_update_status(task, TaskStatus.COMPLETED)
successful_task(task)
def task_pending(task):
_update_status(task, TaskStatus.PENDING)
successful_task(task)
def task_canceled(task):
_update_status(task, TaskStatus.CANCELED)
process_task(task)
def successful_task(task):
process_task(task)
send_notification(task)
def process_task(task):
assign_user(task)
notify_user(task)
cleanup(task)
def _update_status(task, status):
task.status = status
task.save(update_fields=['status'])
I have written the following tests:
def test_task_completed(mocker, task):
mock_successful_task = mocker.patch('services.successful_task')
task_completed(task)
assert task.status == TaskStatus.COMPLETED
mock_successful_task.called_once_with(task)
def test_task_pending(mocker, task):
mock_successful_task = mocker.patch('services.successful_task')
task_pending(task)
assert task.status == TaskStatus.PENDING
mock_successful_task.called_once_with(task)
def test_task_canceled(mocker, task):
mock_process_task = mocker.patch('services.process_task')
task_pending(task)
assert task.status == TaskStatus.CANCELED
mock_process_task.called_once_with(task)
def test_successful_task(mocker, task):
mock_process_task = mocker.patch('services.process_task')
mock_send_notification = mocker.patch('notifications.send_notification')
mock_process_task.called_once_with(task)
mock_send_notification.called_once_with(task)
def test_process_task(mocker, task):
mock_assign_user = mocker.patch('users.assign_user')
mock_notify_user = mocker.patch('notifications.notify_user')
mock_cleanup = mocker.patch('utils.cleanup')
mock_assign_user.called_once_with(task)
mock_notify_user.called_once_with(task)
mock_cleanup.called_once_with(task)
As you can see some tests like test_successful_task and test_process_task are just testing if specific functions are called.
But does it make sense to write a test for this or do I understand something wrong and my unit tests are just bad? I don't know another solution how I should test these functions.
In my experience, tests like these are very brittle because they depend on implementation details. A unit test should only be concerned with the results of the method being tested. Ideally, this means asserting against the return value. If there are side effects, you can assert those instead. But then you should probably look at those side effects and find a different solution that doesn't require them.
I would say no, they're not useful.
Unit tests are supposed to test functionality, here's some input, I call this, here's my result, does it match what I expect? Things needs to be clear and verifiable.
When you have a test which verifies a method has been called, what do you really have?
Pure uncertainty. Ok, a thing has been called, but how is that useful? You are not verifying a result, the method you're calling can do a million things and you have no idea what it does.
Code calling a method is an implementation detail and your unit tests are not supposed to have that kind of knowledge.
Why do we write unit tests?
- to check functionality
- to help refactoring
If you need to change your tests every time your code changes then you haven't really accomplished one of the main reasons for unit testing.
If your code changes and that method is not called anymore, then what?
You now have to go and change the test? Change it to what though? If your next move is to remove the test then why did you have in the first place?
What if someone else has to deal with this issue, 6 months down the road? There is no documentation to check and see why is there a test checking a method has been called?
Bottom line, a test like this has zero value and all it does is introduce uncertainty.
White box tests can be useful to detect some regression or to assert that a specific action is made.
For example you can verify that you don't interact with your DB in this particular case or that you've correctly called the notification service.
However, the drawback is that you're likely to change the test when you change the code, because you're test is very tied to the implementation.
This can be painful when you are refactoring, because you also need to refactor the test. You could forget an assertion or a step and create a false positive test with a regression.
I would use it only if it makes sens and if you need it to assert what's going on in details.
You can search on the web TDD: London vs Detroit.
You'll find interesting stuff.
This is not exactly the purpose of unit tests, though it does have uses. Unit tests aim to improve the quality of the code by testing functionality and results-- it would be more beneficial to write unit tests to test the functionality of each method called.
With that being said, if you have one function that calls 4 other functions and you want to check if they actually get executed in your main block of code, then this makes sense. But you should definitely be writing unit tests for your submethods as well.
Yes, it makes sense. However, I would look at unittest.mock.Mock.assert_called_with
In my experience, yes.
When you design a test, you know that you have to deal with 4 elements
pre-conditions (context)
inputs
outputs
post-conditions (side effects)
We all agree that it would be more easy to test and code if the beahviour of the function under test depended only from inputs and outputs, but in some case this cannot happen, especially when your logic deals with I/O and/or its goal is to issue a mutation of the application state. This means that your test have to be aware of post-conditions. But what can make us sure that a post-condition is met?
pick this method
public class UserService
{
public void addUser(User toAdd)
}
this method adds a user on the database; in a more elegant way, we can say that adds the User to a collection, which is abstracted by the repository semantic. So the side effect of the method is that a userRepository.save(User user) is called. You can mock this method and expect that has been called once with given argument, or make the test fail
Obiviously this can be achivied only if the method has been mocked, so the test is not influenced from the behaviour of a unit that is not under test.
I recognize that the drawback is to make the test brittle, but at the same time
in tdd, it makes fail tests that don't call the mocked function, so the test states "hey, the addUser relies upon the UserRepository.save()!" if you're into this style
The test will be broken if the dependent function interface changes, but we don't want to change interface often, am I right?
before to add dependencies to your method you will think twice, this is an hint to write cleaner code
I'm adding some (epydoc) documentation to a package I've written, and I'm coming across a lot of instances where I'm repeating myself a multitude of times.
def script_running(self, script):
"""Return if script is running
#param script: Script to check whether running
#return: B{True} if script is running, B{False} otherwise
#rtype: C{bool}
"""
PEP257 says that:
One-liners are for really obvious cases.
and also
The docstring for a function or method should summarize its behavior and document its arguments, return value(s), side effects, exceptions raised, and restrictions on when it can be called (all if applicable).
Is there a general guideline or standard practice for when to draw the line between a one-liner (description) and full param/return fields?
Or when generating documentation should I include every applicable field for each function, regardless of how repetitive it seems?
Bonus question: Syntactically, what's the best way to describe the script param?
The general guideline you are looking for is right in PEP257 in what you quoted, maybe you just need to see it in action.
Your function is a good candidate for a one-line docstring ("really obvious cases"):
def script_running(self, script):
"""Check if the script is running."""
Usually if you say that a function is checking something it means that it's going to return True or False, but if you like you could be more specific:
def script_running(self, script):
"""Return True if the script is running, False otherwise."""
Once again all in one line.
I would probably also change the name of your function, because there's no need to emphasize on what the function works in its name (a script). A function name should be something sweet, short and meaningful about what the function does. Probably I'd go with:
def check_running(self, script):
"""Return True if the script is running, False otherwise."""
Sometimes the function-name-imagination is tired by all the coding, but you should try anyway to do your best.
For a multiline example, let me borrow a docstring from the google guidelines:
def fetch_bigtable_rows(big_table, keys, other_silly_variable=None):
"""Fetches rows from a Bigtable.
Retrieves rows pertaining to the given keys from the Table instance
represented by big_table. Silly things may happen if
other_silly_variable is not None.
Args:
big_table: An open Bigtable Table instance.
keys: A sequence of strings representing the key of each table row
to fetch.
other_silly_variable: Another optional variable, that has a much
longer name than the other args, and which does nothing.
Returns:
A dict mapping keys to the corresponding table row data
fetched. Each row is represented as a tuple of strings. For
example:
{'Serak': ('Rigel VII', 'Preparer'),
'Zim': ('Irk', 'Invader'),
'Lrrr': ('Omicron Persei 8', 'Emperor')}
If a key from the keys argument is missing from the dictionary,
then that row was not found in the table.
Raises:
IOError: An error occurred accessing the bigtable.Table object.
"""
This could be one way to "summarize its behavior and document its arguments, return value(s), side effects, exceptions raised, and restrictions on when it can be called (all if applicable)".
You might also be interested to look at this example of pypi project that it's meant to be documented with Sphinx.
My 2 cents: Guidelines are meant to give you an idea about what you should and shouldn't do, but they are not strict rules that you have to blindly follow. So at the end choose what you feel to be better.
I would like to clear something that is been said in another answer about hitting the Maximum Line Length with a docstring.
PEP8 tells you to "Limit all lines to a maximum of 79 characters" even though at the end everyone does 80.
This are 80 characters:
--------------------------------------------------------------------------------
And this may be an edge case where a little long one sentence is all you really need:
def my_long_doc_function(arg1, arg2):
"""This docstring is long, it's a little looonger than the 80 characters
limit.
"""
Is like a one-line docstring, meaning that is for really obvious cases, but on your editor (with the 80 character limit) is on multiple lines.
I think there is likely always some degree of repetition involved when adding extended syntax for docstrings, i.e. epydoc/sphinx markup.
I would also say this matter is subjective rahter than objective. Explicit is better than implicit, and would seem to follow the Zen of Python more.
I am looking into the unittest package, and I'm not sure of the proper way to structure my test cases when writing a lot of them for the same method. Say I have a fact function which calculates the factorial of a number; would this testing file be OK?
import unittest
class functions_tester(unittest.TestCase):
def test_fact_1(self):
self.assertEqual(1, fact(1))
def test_fact_2(self):
self.assertEqual(2, fact(2))
def test_fact_3(self):
self.assertEqual(6, fact(3))
def test_fact_4(self):
self.assertEqual(24, fact(4))
def test_fact_5(self):
self.assertFalse(1==fact(5))
def test_fact_6(self):
self.assertRaises(RuntimeError, fact, -1)
#fact(-1)
if __name__ == "__main__":
unittest.main()
It seems sloppy to have so many test methods for one method. I'd like to just have one testing method and put a ton of basic test cases (ie 4! ==24, 3!==6, 5!==120, and so on), but unittest doesn't let you do that.
What is the best way to structure a testing file in this scenario?
Thanks in advance for the help.
You can put the asserts in a loop:
def test_fact(self):
tests = [(1,1), (2,2), (3,6), (4,24), (5,120)]
for n,f in tests:
self.assertEqual(fact(n), f)
I'd say your ways of doing it is generally fine (but read on).
You could, as interjay suggested, do a loop (and, incidentally, it only counts as one test because the unittest module counts the number of functions, not the number of asserts). But I assume you won't exhaustively try to test every number, or even all numbers within a very large interval. So looping won't save you much, and, especially in testing, you should aim at being explicit.
Having said that, you should test for a small number of subsequent numbers (say, 1 through 5), and then try to get a feel for where possible corner cases and failures points are. Say, test for 10, 100, 1000 (that is, change order of magnitude), negative numbers, zero etc.
BTW, watch out for your two last tests. The first of them doesn't mean much. fact(5) is different than a LOT of numbers (infinite numbers, actually). Test for the correct case, testing for the incorrect ones isn't productive.
def test_fact_5(self):
self.assertFalse(1==fact(5))
The second one is badly named: "test_fact_6" makes me think you're testing fact(6). You should name it as something like "test_fact_minus_one", or at least "test_fact_negative_number".
def test_fact_6(self):
self.assertRaises(RuntimeError, fact, -1)
Test naming is very important, both when you're debugging errors and when you refer back to the tests as documentation.
You've said you're looking into unittest, but consider using nose tests instead - they allow you to generate independant test cases programmatically.
How to generate dynamic (parametrized) unit tests in python? (answer)