I'm a bit of a beginner in terms of unit-testing and still trying to get familiar with some of the things. This is part of my own project and I'm stuck on how to test it fully.
There was a question in this direction already, but it did only concern itself on how to repeatedly ask the user for input, not on how to unittest it.
Goal:
I have a function that asks the user for input and repeats the request if the input is invalid. My goal is to figure out how to test if the input-request is repeated if the user gives invalid input. By that I mean, I'm trying to test whether the mechanism to repeat the input-request under the defined circumstances works as intended.
The Code:
The function asks for a regular expression from the user and compiles it into an SRE_Pattern object from the re-package (python standard library). If the user provides an input and it's not a valid expression, the input-request is repeated.
import re
def request_regex_pattern(input_message):
regex_pattern = None
while True:
regex = input(input_message)
if not regex:
print("No Regex provided.")
break
try:
regex_pattern = re.compile(regex, re.IGNORECASE)
break
except re.error:
print("The input was not valid regular expression")
continue
return regex_pattern
Tests so far:
What I can test so far is whether for valid input (e.g. \d\d) I get correct output (SRE_Patternobject of that regular expression) using mocking.
import unittest as ut
from unittest import mock
import re
class TestUserInput(ut.TestCase):
def test_request_regex_pattern(self):
with mock.patch('builtins.input', return_value='\d\d'):
test_pattern = request_regex_pattern('')
test_string = '01'
self.assertIsNotNone(test_pattern.match(test_string))
I've thought about this and googled for a while now but couldn't come to a satisfying answer.
Is there a sensible way to test whether the input-request was repeated?
What are the best-practices there?
Using python's default unittest library is not mandatory for solutions. However, generally solutions using the standard libraries would be preferred as that reduces the number of requirements needed for the project I'm working on.
Thank you very much for your time !
MrBreanBremen pointed me in the right direction to linking me to his answer and kindly helped me with some of my follow-up questions regarding syntax and understanding the principle. I want to elaborate and dumb down the principles behind what he used there a bit for easier understanding.
The Principle
The way to check this is by patching a function that is called under the circumstances that you want to check. This replaces them with MagicMock-objects that know when they have been called and with which parameters !
Once the function is patched, you can then use that MagicMocks assert methods such as assert_called_with(), assert_called_once(), assert_called() and/or assert_not_called() to check whether the function that this MagicMock object replaced was called at all and with the parameters you were expecting.
Now how does this apply to this question?
First back to what is our problem. We have 3 Test-cases:
1) User provides valid regular expression that can compile into an SRE_Pattern object --> Returns an SRE_Pattern object
2) User provides no input (just hits enter) --> Returns None
3) User provide input that can't be compiled into an SRE_Pattern object (invalid input), triggering the print("The input was not valid regular expression") statement --> Never Returns anything
We only care about circumstances of 3), as case 1) and 2) have a defined output that we can check easily with "normal" unit-tests, while case 3) explicitly can't output anything, leading us to our problem.
As already hinted at in 3), under these circumstances only the print function is called. Meaning that for our test we should patch it, so that we can receive a MagicMock-object for this function and use its assert_called_with() together with the string that print-statement gets, as that string only occurs in that section of code. It is "unique" for these circumstances !
We have one more issue to solve though. Once we patch the builtins.input as before, but with something that causes our while-loop to repeat, we will still be stuck in a while-loop! The function call request_regex_pattern() will never end ! Since we can't exit the function normally, we'll have to exit by causing an Exception. Thus, we need to also patch that into one of the functions that is called when these circumstances happen. In this case, we can conveniently add this side-effect to our patch of print. We can then catch that Exception with a context-manager with self.assertRaises(Exception) to keep our test from failing.
The code:
import unittest as ut
from unittest import mock
import re
class TestUserInput(ut.TestCase):
def test_request_regex_pattern_non_regex_input(self):
with mock.patch('builtins.input', return_value='\l\d'):
with mock.patch('builtins.print', side_effect=[None, Exception('To Break the Loop!')]) as mocked_print:
with self.assertRaises(Exception):
ui.request_regex_pattern('')
mocked_print.assert_called_with('The input was not valid regular expression')
Improving Readability
This was kept with the "patch through context-manager"-syntax for consistency with the previously displayed unit-test.
As you can see, that code is not nice to read due to all the context-managers. As such, it is better to use the patch-decorator instead, as MrBreanBremen did! This will then pass the MagicMock-objects for these functions as parameters to your test, in the order that the patches are applied. Here mocked_input is the MagicMock object of the patched input() method and mocked_print is the MagicMock object of the patched print() method.
import unittest as ut
from unittest import mock
import re
class TestUserInput(ut.TestCase):
#mock.patch('builtins.input', return_value='\l\d')
#mock.patch('builtins.print', side_effect=[None, Exception('To Break the Loop!')])
def test_request_regex_pattern_non_regex_input(self, mocked_input, mocked_print):
with self.assertRaises(Exception):
request_regex_pattern('')
mocked_print.assert_called_with('The input was not valid regular expression')
A reliable way to prevent the user from entering the same input more than once would be to simply use a python built-in list. You can use a pre-determined size and just store that many elements inside it. You can append elements to the list and then if the predetermined size is exceeded pop elements from the front (oldest elements). This way you would be able to make sure that the user wouldn't be able to enter the same input as the last N (size of the list) inputs.
Related
I am adding some tests to existing not so test friendly code, as title suggest, I need to test if the complex method actually calls another method, eg.
class SomeView(...):
def verify_permission(self, ...):
# some logic to verify permission
...
def get(self, ...):
# some codes here I am not interested in this test case
...
if some condition:
self.verify_permission(...)
# some other codes here I am not interested in this test case
...
I need to write some test cases to verify self.verify_permission is called when condition is met.
Do I need to mock all the way to the point of where self.verify_permission is executed? Or I need to refactor the def get() function to abstract out the code to become more test friendly?
There are a number of points made in the comments that I strongly disagree with, but to your actual question first.
This is a very common scenario. The suggested approach with the standard library's unittest package is to utilize the Mock.assert_called... methods.
I added some fake logic to your example code, just so that we can actually test it.
code.py
class SomeView:
def verify_permission(self, arg: str) -> None:
# some logic to verify permission
print(self, f"verify_permission({arg=}=")
def get(self, arg: int) -> int:
# some codes here I am not interested in this test case
...
some_condition = True if arg % 2 == 0 else False
...
if some_condition:
self.verify_permission(str(arg))
# some other codes here I am not interested in this test case
...
return arg * 2
test.py
from unittest import TestCase
from unittest.mock import MagicMock, patch
from . import code
class SomeViewTestCase(TestCase):
def test_verify_permission(self) -> None:
...
#patch.object(code.SomeView, "verify_permission")
def test_get(self, mock_verify_permission: MagicMock) -> None:
obj = code.SomeView()
# Odd `arg`:
arg, expected_output = 3, 6
output = obj.get(arg)
self.assertEqual(expected_output, output)
mock_verify_permission.assert_not_called()
# Even `arg`:
arg, expected_output = 2, 4
output = obj.get(arg)
self.assertEqual(expected_output, output)
mock_verify_permission.assert_called_once_with(str(arg))
You use a patch variant as a decorator to inject a MagicMock instance to replace the actual verify_permission method for the duration of the entire test method. In this example that method has no return value, just a side effect (the print). Thus, we just need to check if it was called under the correct conditions.
In the example, the condition depends directly on the arg passed to get, but this will obviously be different in your actual use case. But this can always be adapted. Since the fake example of get has exactly two branches, the test method calls it twice to traverse both of them.
When doing unit tests, you should always isolate the unit (i.e. function) under testing from all your other functions. That means, if your get method calls other methods of SomeView or any other functions you wrote yourself, those should be mocked out during test_get.
You want your test of get to be completely agnostic to the logic inside verify_permission or any other of your functions used inside get. Those are tested separately. You assume they work "as advertised" for the duration of test_get and by replacing them with Mock instances you control exactly how they behave in relation to get.
Note that the point about mocking out "network requests" and the like is completely unrelated. That is an entirely different but equally valid use of mocking.
Basically, you 1.) always mock your own functions and 2.) usually mock external/built-in functions with side effects (like e.g. network or disk I/O). That is it.
Also, writing tests for existing code absolutely has value. Of course it is better to write tests alongside your code. But sometimes you are just put in charge of maintaining a bunch of existing code that has no tests. If you want/can/are allowed to, you can refactor the existing code and write your tests in sync with that. But if not, it is still better to add tests retroactively than to have no tests at all for that code.
And if you write your unit tests properly, they still do their job, if you or someone else later decides to change something about the code. If the change breaks your tests, you'll notice.
As for the exception hack to interrupt the tested method early... Sure, if you want. It's lazy and calls into question the whole point of writing tests, but you do you.
No, seriously, that is a horrible approach. Why on earth would you test just part of a function? If you are already writing a test for it, you may as well cover it to the end. And if it is so complex that it has dozens of branches and/or calls 10 or 20 other custom functions, then yes, you should definitely refactor it.
Does Python has a feature that allows one to evaluate a function or expression and if the evaluation fails (an exception is raised) return a default value.
Pseudo-code:
evaluator(function/expression, default_value)
The evaluator will try to execute the function or expression and return the result is the execution is successful, otherwise the default_value is returned.
I know I create a user defined function using try and except to achieve this but I want to know if the batteries are already included before going off and creating a custom solution.
In order to reuse code, you can create a decorating function (that accepts a default value) and decorate your functions with it:
def handle_exceptions(default):
def wrap(f):
def inner(*a):
try:
return f(*a)
except Exception, e:
return default
return inner
return wrap
Now let's see an example:
#handle_exceptions("Invalid Argument")
def test(num):
return 15/num
#handle_exceptions("Input should be Strings only!")
def test2(s1, s2):
return s2 in s1
print test(0) # "Invalid Argument"
print test(15) # 1
print test2("abc", "b") # True
print test2("abc", 1) # Input should be Strings only!
No, the standard way to do this is with try... except.
There is no mechanism to hide or suppress any generic exception within a function. I suspect many Python users would consider indiscriminate use of such a function to be un-Pythonic for a couple reasons:
It hides information about what particular exception occurred. (You might not want to handle all exceptions, since some could come from other libraries and indicate conditions that your program can't recover from, like running out of disk space.)
It hides the fact that an exception occurred at all; the default value returned in case of an exception might coincide with a valid non-default value. (Sometimes reasonable, sometimes not really so.)
One of the principles of the Pythonic philosophy, I believe, is that "explicit is better than implicit," so Python generally avoids automatic type casting and error recovery, which are features of more "implicit- friendly"languages like Perl.
Although the try... except form can be a bit verbose, in my opinion it has a lot of advantages in terms of clearly showing where an exception may occur and what the control flow is around that exception.
I'm trying to find a similar approach to Qunit's assertions in Python. When using assertions in Qunit, the message parameter is used in a very descriptive fashion.
test( "test", function() {
ok( fn([])==None, "Function should return 0 if no users" );
ok( fn(["Test User"])==1, "Function should return 1 is users supplied" );
});
Python's unittest module on the other hand, uses the message parameter is a somewhat more negative context. These are only shown when an assertion fails.
class TestSequenceFunctions(unittest.TestCase):
def test_choice(self):
seq = range(10)
element = random.choice(seq)
self.assertTrue(element in seq, msg="Element not found in sequence")
The end result of the Qunit is that there is much clearer transcript which could be compared against a spec document.
I realise that in Python, a similar approach would be achieved by perhaps say writing
def test_choice_ensure_element_exists_in_sequence(self):
It's not the same though. The output isn't presented in a nice way, and test lifecycle then performs setup and teardown for each label you want to use, which isn't necessarily what you want.
There might be a library out there which takes this approach, so perhaps this issue is already solved. Neither the python unittest library or pytest appear to work in this fashion though.
Your problem could be simply that don't know the unittest libary well enough yet. I find being able to write
self.assertIn('s', (1,3,4))
To be very short, expressive and readable.
And if you use the correct assertion method on the testcase then you rarely need to add your own message. assertIn has a perfectly reasonable output all by itself:
AssertionError: 's' not found in (1, 3, 4)
So rather than writing heaps of comments/message code. I rely on well named assertions combined with helpful default messages. If a well named assertion and helpful error message has not already been provided then I extend the test case and add my own.
self.assert_user_is_administrator(user)
Is very readable and will have a nice message if it fails that I provided in only one location.
This one's a structure design problem, I guess. Back for some advice.
To start: I'm writing a module. Hence the effort of making it as usable to potential developers as possible.
Inside an object (let's call it Swoosh) I have a method which, when called, may result in either success (a new object is returned -- for insight: it's an httplib.HTTPResponse) or failure (surprising, isn't it?).
I'm having trouble deciding how to handle failures. There are two main cases here:
user supplied data that was incorrect
data was okay, but user interaction will be needed () - I need to pass back to the user a string that he or she will need to use in some way.
In (1) I decided to raise ValueError() with an appropriate description.
In (2), as I need to actually pass a str back to the user.. I'm not sure about whether it would be best to just return a string and leave it to the user to check what the function returned (httplib.HTTPResponse or str) or raise a custom exception? Is passing data through raising exceptions a good idea? I don't think I've seen this done anywhere, but on the other hand - I haven't seen much.
What would you, as a developer, expect from an object/function like this?
Or perhaps you find the whole design ridiculous - let me know, I'll happily learn.
As much as I like the approach of handling both cases with specifically-typed exceptions, I'm going to offer a different approach in case it helps: callbacks.
Callbacks tend to work better if you're already using an asynchronous framework like Twisted, but that's not their only place. So you might have a method that takes a function for each outcome, like this:
def do_request(on_success, on_interaction_needed, on_failure):
"""
Submits the swoosh request, and awaits a response.
If no user interaction is needed, calls on_success with a
httplib.HTTPResponse object.
If user interaction is needed, on_interaction_needed is
called with a single string parameter.
If the request failed, a ValueError is passed to on_failure
"""
response = sumbit_request()
if response.is_fine():
on_success(response)
elif response.is_partial()
on_interaction_needed(response.message)
else:
on_failure(ValueError(response.message))
Being Python, there are a million ways to do this. You might not like passing an exception to a function, so you maybe just take a callback for the user input scenario. Also, you might pass the callbacks in to the Swoosh initialiser instead.
But there are drawbacks to this too, such as:
Carelessness may result in spaghetti code
You're allowing your caller to inject logic into your function (eg. exceptions raised in the callback will propagate out of Swoosh)
My example here is simple, your actual function might not be
As usual, careful consideration and good documentation should avoid these problems. In theory.
I think raising an exception may actually be a pretty good idea in this case. Squashing multiple signals into a single return value of a function isn't ideal in Python, due to duck typing. It's not very Pythonic; every time you need to do something like:
result = some_function(...)
if isinstance(result, TypeA):
do_something(result)
elif isinstance(result, TypeB):
do_something_else(result)
you should be thinking about whether it's really the best design (as you're doing).
In this case, if you implement a custom exception, then the code that calls your function can just treat the returned value as a HTTPResponse. Any path where the function is unable to return something its caller can treat that way is handled by throwing an exception.
Likewise, the code that catches the exception and prompts the user with the message doesn't have to worry about the exact type of the thing its getting. It just knows that it's been explicitly instructed (by the exception) to show something to the user.
If the user interaction case means the calling code has to show a prompt, get some input and them pass control back to your function, it might be ugly trying to handle that with an exception. Eg,
try:
Swoosh.method()
except UserInteraction, ex:
# do some user interaction stuff
# pass it back to Swoosh.method()?
# did Swoosh need to save some state from the last call?
except ValueError:
pass # whatever
If this user interaction is a normal part of the control flow, it might be cleaner to pass a user-interaction function into your method in the first place - then it can return a result to the Swoosh code. For example:
# in Swoosh
def method(self, userinteractor):
if more_info_needed:
more_info = userinteractor.prompt("more info")
...
ui = MyUserInteractor(self) # or other state
Swoosh.method(ui)
You can return a tuple of (httplib.HTTPResponse, str) with the str being optionally None.
Definitely raise an exception for 1).
If you don't like returning a tuple, you can also create a "response object" i.e. an instance of a new class ( lets say SomethingResponse ) that encapsulates the HTTPResponse with optional messages to the end-user( in the simplest case, just a str).
I'm trying to write a freeze decorator for Python.
The idea is as follows :
(In response to the two comments)
I might be wrong but I think there is two main use of
test case.
One is the test-driven development :
Ideally , developers are writing case before writing the code.
It usually helps defining the architecture because this discipline
forces to define the real interfaces before development.
One may even consider that in some case the person who
dispatches job between dev is writing the test case and
use it to illustrate efficiently the specification he has in mind.
I don't have any experience of the use of test case like that.
The second is the idea that all project with a decent
size and a several programmers is suffering from broken code.
Something that use to work may get broken from a change
that looked like an innocent refactoring.
Though good architecture, loose couple between component may
help to fight against this phenomenon ; you will sleep better
at night if you have written some test case to make sure
that nothing will break your program's behavior.
HOWEVER,
Nobody can deny the overhead of writting test cases. In the
first case one may argue that test case is actually guiding
development and is therefore not to be considered as an overhead.
Frankly speaking, I'm a pretty young programmer and if I were
you, my word on this subject is not really valuable...
Anyway, I think that mosts company/projects are not working
like that, and that unit tests are mainly used in the second
case...
In other words, rather than ensuring that the program is
working correctly, it is aiming at checking that it will
work the same in the future.
This needs can be met without the cost of writing tests,
by using this freezing decorator.
Let's say you have a function
def pow(n,k):
if n == 0: return 1
else: return n * pow(n,k-1)
It is perfectly nice, and you want to rewrite it as an optimized version.
It is part of a big project. You want it to give back the same result
for a few value.
Rather than going through the pain of test cases, one could use some
kind of freeze decorator.
Something such that the first time the decorator is run,
the decorator run the function with the defined args (below 0, and 7)
and saves the result in a map ( f --> args --> result )
#freeze(2,0)
#freeze(1,3)
#freeze(3,5)
#freeze(0,0)
def pow(n,k):
if n == 0: return 1
else: return n * pow(n,k-1)
Next time the program is executed, the decorator will load this map and check
that the result of this function for these args as not changed.
I already wrote quickly the decorator (see below), but hurt a few problems about
which I need your advise...
from __future__ import with_statement
from collections import defaultdict
from types import GeneratorType
import cPickle
def __id_from_function(f):
return ".".join([f.__module__, f.__name__])
def generator_firsts(g, N=100):
try:
if N==0:
return []
else:
return [g.next()] + generator_firsts(g, N-1)
except StopIteration :
return []
def __post_process(v):
specialized_postprocess = [
(GeneratorType, generator_firsts),
(Exception, str),
]
try:
val_mro = v.__class__.mro()
for ( ancestor, specialized ) in specialized_postprocess:
if ancestor in val_mro:
return specialized(v)
raise ""
except:
print "Cannot accept this as a value"
return None
def __eval_function(f):
def aux(args, kargs):
try:
return ( True, __post_process( f(*args, **kargs) ) )
except Exception, e:
return ( False, __post_process(e) )
return aux
def __compare_behavior(f, past_records):
for (args, kargs, result) in past_records:
assert __eval_function(f)(args,kargs) == result
def __record_behavior(f, past_records, args, kargs):
registered_args = [ (a, k) for (a, k, r) in past_records ]
if (args, kargs) not in registered_args:
res = __eval_function(f)(args, kargs)
past_records.append( (args, kargs, res) )
def __open_frz():
try:
with open(".frz", "r") as __open_frz:
return cPickle.load(__open_frz)
except:
return defaultdict(list)
def __save_frz(past_records):
with open(".frz", "w") as __open_frz:
return cPickle.dump(past_records, __open_frz)
def freeze_behavior(*args, **kvargs):
def freeze_decorator(f):
past_records = __open_frz()
f_id = __id_from_function(f)
f_past_records = past_records[f_id]
__compare_behavior(f, f_past_records)
__record_behavior(f, f_past_records, args, kvargs)
__save_frz(past_records)
return f
return freeze_decorator
Dumping and Comparing of results is not trivial for all type. Right now I'm thinking about using a function (I call it postprocess here), to solve this problem.
Basically instead of storing res I store postprocess(res) and I compare postprocess(res1)==postprocess(res2), instead of comparing res1 res2.
It is important to let the user overload the predefined postprocess function.
My first question is :
Do you know a way to check if an object is dumpable or not?
Defining a key for the function decorated is a pain. In the following snippets
I am using the function module and its name.
** Can you think of a smarter way to do that. **
The snippets below is kind of working, but opens and close the file when testing and when recording. This is just a stupid prototype... but do you know a nice way to open the file, process the decorator for all function, close the file...
I intend to add some functionalities to this. For instance, add the possibity to define
an iterable to browse a set of argument, record arguments from real use, etc.
Why would you expect from such a decorator?
In general, would you use such a feature, knowing its limitation... Especially when trying to use it with POO?
"In general, would you use such a feature, knowing its limitation...?"
Frankly speaking -- never.
There are no circumstances under which I would "freeze" results of a function in this way.
The use case appears to be based on two wrong ideas: (1) that unit testing is either hard or complex or expensive; and (2) it could be simpler to write the code, "freeze" the results and somehow use the frozen results for refactoring. This isn't helpful. Indeed, the very real possibility of freezing wrong answers makes this a bad idea.
First, on "consistency vs. correctness". This is easier to preserve with a simple mapping than with a complex set of decorators.
Do this instead of writing a freeze decorator.
print "frozen_f=", dict( (i,f(i)) for i in range(100) )
The dictionary object that's created will work perfectly as a frozen result set. No decorator. No complexity to speak of.
Second, on "unit testing".
The point of a unit test is not to "freeze" some random results. The point of a unit test is to compare real results with results developed another (simpler, more obvious, poorly-performing way). Usually unit tests compare hand-developed results. Other times unit tests use obvious but horribly slow algorithms to produce a few key results.
The point of having test data around is not that it's a "frozen" result. The point of having test data is that it is an independent result. Done differently -- sometimes by different people -- that confirms that the function works.
Sorry. This appears to me to be a bad idea; it looks like it subverts the intent of unit testing.
"HOWEVER, Nobody can deny the overhead of writting test cases"
Actually, many folks would deny the "overhead". It isn't "overhead" in the sense of wasted time and effort. For some of us, unittests are essential. Without them, the code may work, but only by accident. With them, we have ample evidence that it actually works; and the specific cases for which it works.
Are you looking to implement invariants or post conditions?
You should specify the result explicitly, this wil remove most of you problems.