Whenever pytest is outputting the tests that it has, it always prints out the parameter values, but never the names of the variables.
For example:
PASSED test_create_patient.py::test_create_patient[False-True]
What I would like it to output is:
PASSED test_create_patient.py::test_create_patient[arg1=False-arg2=True]
Is this possible? I have tried various pytest flags like -vvv but nothing seems to work for this. It would be very helpful for debugging purposes.
I've looked into answers like this https://stackoverflow.com/a/18496751/8903959 but I don't understand them.
Generally, you can use the ids parameter in pytest.mark.parametrize to customize the parameter ids. In your case, this is inconvenient. While ids allows you to provide a custom function to adapt the output, this function only gets the parameter value as argument, not the parameter name, so you cannot use it to compose your wanted output. Instead, you would have to provide a list of ids:
#pytest.mark.parametrize("arg1, arg2", [(False, True), (True, False)],
ids=["arg1=False-arg2=True",
"arg1=True-arg2=False"])
def test_create_patient(arg1, arg2):
pass
To avoid this, you can use the hook function pytest_make_parametrize_id instead, which gets both the parameter name and the value. You have to put it in a conftest.py visible to your test. In your case, you can write:
conftest.py
def pytest_make_parametrize_id(config, val, argname):
return f"{argname}={val}"
and you would get the wanted notation for each test.
If you want to have this notation only for verbose output (say, with the option -vv or more), you can use config.option to check the verbosity and adapt the id accordingly:
conftest.py
def pytest_make_parametrize_id(config, val, argname):
if config.option.verbose >= 2: # -vv or -vvv
return f"{argname}={val}"
return repr(val) # the default
Related
I am adding some tests to existing not so test friendly code, as title suggest, I need to test if the complex method actually calls another method, eg.
class SomeView(...):
def verify_permission(self, ...):
# some logic to verify permission
...
def get(self, ...):
# some codes here I am not interested in this test case
...
if some condition:
self.verify_permission(...)
# some other codes here I am not interested in this test case
...
I need to write some test cases to verify self.verify_permission is called when condition is met.
Do I need to mock all the way to the point of where self.verify_permission is executed? Or I need to refactor the def get() function to abstract out the code to become more test friendly?
There are a number of points made in the comments that I strongly disagree with, but to your actual question first.
This is a very common scenario. The suggested approach with the standard library's unittest package is to utilize the Mock.assert_called... methods.
I added some fake logic to your example code, just so that we can actually test it.
code.py
class SomeView:
def verify_permission(self, arg: str) -> None:
# some logic to verify permission
print(self, f"verify_permission({arg=}=")
def get(self, arg: int) -> int:
# some codes here I am not interested in this test case
...
some_condition = True if arg % 2 == 0 else False
...
if some_condition:
self.verify_permission(str(arg))
# some other codes here I am not interested in this test case
...
return arg * 2
test.py
from unittest import TestCase
from unittest.mock import MagicMock, patch
from . import code
class SomeViewTestCase(TestCase):
def test_verify_permission(self) -> None:
...
#patch.object(code.SomeView, "verify_permission")
def test_get(self, mock_verify_permission: MagicMock) -> None:
obj = code.SomeView()
# Odd `arg`:
arg, expected_output = 3, 6
output = obj.get(arg)
self.assertEqual(expected_output, output)
mock_verify_permission.assert_not_called()
# Even `arg`:
arg, expected_output = 2, 4
output = obj.get(arg)
self.assertEqual(expected_output, output)
mock_verify_permission.assert_called_once_with(str(arg))
You use a patch variant as a decorator to inject a MagicMock instance to replace the actual verify_permission method for the duration of the entire test method. In this example that method has no return value, just a side effect (the print). Thus, we just need to check if it was called under the correct conditions.
In the example, the condition depends directly on the arg passed to get, but this will obviously be different in your actual use case. But this can always be adapted. Since the fake example of get has exactly two branches, the test method calls it twice to traverse both of them.
When doing unit tests, you should always isolate the unit (i.e. function) under testing from all your other functions. That means, if your get method calls other methods of SomeView or any other functions you wrote yourself, those should be mocked out during test_get.
You want your test of get to be completely agnostic to the logic inside verify_permission or any other of your functions used inside get. Those are tested separately. You assume they work "as advertised" for the duration of test_get and by replacing them with Mock instances you control exactly how they behave in relation to get.
Note that the point about mocking out "network requests" and the like is completely unrelated. That is an entirely different but equally valid use of mocking.
Basically, you 1.) always mock your own functions and 2.) usually mock external/built-in functions with side effects (like e.g. network or disk I/O). That is it.
Also, writing tests for existing code absolutely has value. Of course it is better to write tests alongside your code. But sometimes you are just put in charge of maintaining a bunch of existing code that has no tests. If you want/can/are allowed to, you can refactor the existing code and write your tests in sync with that. But if not, it is still better to add tests retroactively than to have no tests at all for that code.
And if you write your unit tests properly, they still do their job, if you or someone else later decides to change something about the code. If the change breaks your tests, you'll notice.
As for the exception hack to interrupt the tested method early... Sure, if you want. It's lazy and calls into question the whole point of writing tests, but you do you.
No, seriously, that is a horrible approach. Why on earth would you test just part of a function? If you are already writing a test for it, you may as well cover it to the end. And if it is so complex that it has dozens of branches and/or calls 10 or 20 other custom functions, then yes, you should definitely refactor it.
I am currently developing an application (my_app) that uses logic from another python package (internal_package), that is being developed in parallel.
Lets say I use a function func1 from internal_package in my code. In internal_package, it is defined like this:
def func1(a, b):
# do something
...
However, If the function signature of func1 (i.e. the parameters with which the function is called) changes, lets say to
def func1(a, b, c):
# do something
...
My code would of course raise an Exception, since I did not pass parameter c.
To become aware of that immediately, I want to write a test in pytest. In my understanding, this kind of test is not a unit test, but an integration test.
My approach would be something like this:
import inspect
import internal_package
def test_func1_signature():
expected_signature = ? # not sure how to correctly provide the expected signature here
actual_signature = inspect.signature(internal_package.func1)
assert actual_signature == expected_signature
I would expect this to be a common issue, but I was not able to find anything related.
What would be the most pythonic way of performing this sort of test?
In case anyone is interested, I ended up using getfullargspec from the inbuilt inspect library. In the documentation it says:
Get the names and default values of a Python function’s parameters. A named tuple is returned:
FullArgSpec(args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations)
As I am interested in in the args part, I retrieve them from func1 as follows:
inspect.getfullargspec(func1)[0]
Which then returns ['a', 'b']. I can then write a simple test in pytest like this:
import pytest
import inspect
def test_func1_signature:
expected_signature = ['a', 'b'] # I specify what i expect
actual_signature = inspect.getfullargspec(func1)[0]
assert actual_signature == expected_signature
If anyone knows a better/more pythonic/more correct way, please let me know.
I'm doing a bunch of unit tests for a bunch of methods I have in my code and started to realize I could really condense this code block down if I could do some sort of macro that would just take care of all my insertions for me.
This is an example of a method I would use, however the only differences is the parameters c_unit_case also c_test_case are based on the methods in lib.
def test_case(filepath):
tests = parsedfile(filepath)
for unittests in tests:
print lib.c_test_case(unittests.params[0])
I'm looking for something sort of like this.
GENERIC_CASE(method_name, params, filepath)
(tests) = parsedfile(filepath)
for unittests in (tests):
args += unittests.(params)
print lib.(method_name)(args)
Is it possible to do this type of thing in Python?
Try something like this and see if this works:
def test_case(method_name, filepath, *args):
tests = parsedfile(filepath)
print (method_name(*args))
*args in the signature of the function lets you put in as many additional arguments as you'd like. Calling method_name(*args) unrolls the arguments into the parameter.
This should handle a variable number of arguments.
This is what it would sort of look like the way you had it:
GENERIC_CASE(method_name, filepath, *params)
(tests) = parsedfile(filepath)
for unittests in (tests):
args += unittests.(*params)
print lib.(method_name)(args)
I'm not 100% if this is what you're looking for.
When I have a parametrized pytest test like in the following case:
#parametrize('repetition', range(3))
#parametrize('name', ['test', '42'])
#parametrize('number', [3,7,54])
def test_example(repetition, name, number):
assert 1 = 1
the test runner prints out lines like follows:
tests\test_example.py:12: test_example[1-test-7]
where the parametrized values show up in the rectangular bracket next to the test functions name (test_example[0]). How can I access the content of the rectangular bracket (i.e. the string 0) inside the test? I have looked at the request
fixture, but could not find a suitable method for my needs.
To be clear: Inside the test method, I want to print the string 1-test-7, which pytest prints on the console.
How can I access the string 1-test-7, which pytest print out during a test?
I do not want to create this string by myself using something like
print str(repetition)+"-"+name+"-"+str(number)
since this would change every time I add a new parametrized argument to the test method.
In addition, if more complex objects are used in the parametrize list (like namedtuple), these objects are just references by a shortcut (e.g. object1, object2, ...).
Addendum: If I use the request fixture as an argument in my test method, I 'see' the string I would like to access when I use the following command
print request.keywords.node.__repr__()
which prints out something like
<Function 'test_example[2-test-3]'>
I am trying to find out how this method __repr__ is defined, in order to access directly the string test_example[2-test-3] from which I easily can extract the string I want, 2-test-3 in this example.
The solution makes use of the built-in request fixture which can be accessed as follows:
def test_example(repetition, name, number, request):
s = request.keywords.node.name
print s[s.find("[")+1:s.find("]")]
which will print the parameter string for each single parametrized test, so each parametrized test can be identified.
I have been doing a lot of searching, and I don't think I've really found what I have been looking for. I will try my best to explain what I am trying to do, and hopefully there is a simple solution, and I'll be glad to have learned something new.
This is ultimately what I am trying to accomplish: Using nosetests, decorate some test cases using the attribute selector plugin, then execute test cases that match a criteria by using the -a switch during commandline invocation. The attribute values for the tests that are executed are then stored in an external location. The command line call I'm using is like below:
nosetests \testpath\ -a attribute='someValue'
I have also created a customized nosetest plugin, which stores the test cases' attributse, and writes them to an external location. The idea is that I can select a batch of tests, and by storing the attributes of these tests, I can do filtering on these results later for reporting purposes. I am accessing the method attributes in my plugin by overriding the "wantMethod" method with the code similar to the following:
def set_attribs(self, method, attribute):
if hasattr(method, attribute):
if not self.method_attributes.has_key(method.__name__):
self.method_attributes[method.__name__] = {}
self.method_attributes[method.__name__][attribute] = getattr(method, attribute)
def wantMethod(self, method):
self.set_attribs(method, "attribute1")
self.set_attribs(method, "attribute2")
pass
I have this working for pretty much all the tests, except for one case, where the test is uing the "yield" keyword. What is happening is that the methods that are generated are being executed fine, but then the method attributes are empty for each of the generated functions.
Below is the example of what I am trying to achieve. The test below retreives a list of values, and for each of those values, yields the results from another function:
#attr(attribute1='someValue', attribute2='anotherValue')
def sample_test_generator(self):
for (key, value) in _input_dictionary.items()
f = partial(self._do_test, key, value)
f.attribute1='someValue'
yield (lambda x: f(), key)
def _do_test(self, input1, input2):
# Some code
From what I have read, and think I understand, when yield is called, it would create a new callable function which then gets executed. I have been trying to figure out how to retain the attribute values from my sample_test_generator method, but I have not been successful. I thought I could create a partial method, and then add the attribute to the method, but no luck. The tests execute without errors at all, it just seems that from my plugin's perspective, the method attributes aren't present, so they don't get recorded.
I realize this a pretty involved question, but I wanted to make sure that the context for what I am trying to achieve is clear. I have been trying to find information that could help me for this particular case, but I feel like I've reached a stumbling block now, so I would really like to ask the experts for some advice.
Thanks.
** Update **
After reading through the feedback and playing around some more, it looks like if I modified the lambda expression, it would achieve what I am looking for. In fact, I didn't even need to create the partial function:
def sample_test_generator(self):
for (key, value) in _input_dictionary.items()
yield (lambda: self._do_test)
The only downside to this approach is that the test name will not change. As I am playing around more, it looks like in nosetests, when a test generator is used, it would actually change the test name in the result based on the keywords it contains. Same thing was happening when I was using the lambda expression with a parameter.
For example:
Using lamdba expression with a parameter:
yield (lambda x: self._do_test, "value1")
In nosetests plugin, when you access the test case name, it would be displayed as "sample_test_generator(value1)
Using lambda expression without a parameter:
yield (lambda: self._do_test)
The test case name in this case would be "sample_test_generator". In my example above, if there are multiple values in the dictionary, then the yield call would occur multiple times. However, the test name would always remain as "sample_test_generator". This is not as bad as when I would get the unique test names, but then not be able to store the attribute values at all. I will keep playing around, but thanks for the feedback so far!
EDIT
I forgot to come back and provide my final update on how I was able to get this to work in the end, there was a little confusion on my part at first, and after I looked through it some more, I figured out that it had to do with how the tests are recognized:
My original implementation assumed that every test that gets picked up for execution goes through the "wantMethod" call from the plugin's base class. This is not true when "yield" is used to generate the test, because at this point, the test method has already passed the "wantMethod" call.
However, once the test case is generated through the "yeild" call, it does go through the "startTest" call from the plug-in base class, and this is where I was finally able to store the attribute successfully.
So in a nut shell, my test execution order looked like this:
nose -> wantMethod(method_name) -> yield -> startTest(yielded_test_name)
In my override of the startTest method, I have the following:
def startTest(self, test):
# If a test is spawned by using the 'yield' keyword, the test names would be the parent test name, appended by the '(' character
# example: If the parent test is "smoke_test", the generated test from yield would be "smoke_test('input')
parent_test_name = test_name.split('(')[0]
if self.method_attributes.has_key(test_name):
self._test_attrib = self.method_attributes[test_name]
elif self.method_attributes.has_key(parent_test_name):
self._test_attrib = self.method_attributes[parent_test_name]
else:
self._test_attrib = None
With this implementation, along with my overide of wantMethod, each test spawned by the parent test case also inherits attributes from the parent method, which is what I needed.
Again, thanks to all who send replies. This was quite a learning experience.
Would this fix your name issue?
def _actual_test(x, y):
assert x == y
def test_yield():
_actual_test.description = "test_yield_%s_%s" % (5, 5)
yield _actual_test, 5, 5
_actual_test.description = "test_yield_%s_%s" % (4, 8) # fail
yield _actual_test, 4, 8
_actual_test.description = "test_yield_%s_%s" % (2, 2)
yield _actual_test, 2, 2
Rename survives #attr too.
does this work?
#attr(attribute1='someValue', attribute2='anotherValue')
def sample_test_generator(self):
def get_f(f, key):
return lambda x: f(), key
for (key, value) in _input_dictionary.items()
f = partial(self._do_test, key, value)
f.attribute1='someValue'
yield get_f(f, key)
def _do_test(self, input1, input2):
# Some code
The Problem ist that the local variables change after you created the lambda.