Let's say I have a test as shown below:
import pytest
import copy
#pytest.fixture(scope='session')
def session_tool(request):
tool = request.config.tool
# Build is the critical part and may fail, raising an exception
tool.build()
return tool
#pytest.fixture
def tool(session_tool):
return copy.deepcopy(session_tool)
def test_tool(tool, args):
assert tool.run(args) == 0
It builds a session-scoped tool and then creates a copy of it for each testcase. But when the build fails, session_tool fixture is executed again for the next testcase, which fails again... until it fails for all testcases. As there are a lot of testcases, it takes some time until the process is finished.
Is there any way to tell pytest to skip all tests which use session_fixture after the first attempt to build fails?
I can think of two approaches:
1) calling pytest.skip() will cause the test to be skipped. This works if it's called from within a fixture as well. In your case, it will cause all the remaining tests to be skipped.
2) calling pytest.exit() will cause your test suite to stop running, as if KeyboardInterrupt was triggered.
Related
How would I unit test the following?
def sigterm_handler(signum, frame):
pid = os.getpid() # type: int
sys.exit(0)
signal.signal(signal.SIGTERM, sigterm_handler)
Should I mock and ensure mock is called?
I would write a test that runs your code in a subprocess which can check if you terminated successfully.
For example, let's say your question code lives in a module called signals.py. You can write a test wrapper module that looks like this:
test_signals_wrapper.py
from time import sleep
from sys import exit
# import this last to ensure it overrides any prior settings
import signals
while True:
sleep(1)
exit(1) # just in case the loop ends for other reasons
Now you can write a unit test that looks like this:
test_signals.py
from subprocess import run, TimeoutExpired
from sys import executable
def test_sigterm_handler():
try:
status = run([executable, '-m', 'test_signals_wrapper'], timeout=30)
except TimeoutExpired:
assert False, 'Did not trigger assertion in 30 seconds'
assert status.returncode == 0, f'Wrong return code: {status.returncode}'
This requires a bit of extra infrastructure for your test, but it solves all the problems with testing this code. By running in a subprocess, you can freely execute sys.exit and get the return value. By having a wrapper script, you can control how the code is loaded and run. You don't need to mock anything, just make sure that your packages are set up correctly, and that your test runner doesn't attempt to pick up the wrapper script as a test.
The code lines you have shown are not suited to be unit-tested, but should rather be integration tested. The reason is, that your code lines consist only of interactions with other components (in this case the signal, sys and os modules).
Therefore, the bugs you can expect to encounter lie in the interactions with these other components: Are you calling the right functions in the right components with the right values for the arguments in the right order and are the results/reactions as you expect them to be?
All these questions can not be answered in a unit-test, where bugs shall be found that can be found in the isolated units: If you mock the signal, sys and/or the os dependencies, then you will write your mocks such that they reflect your (potentially wrong) understanding of these components. The unit-tests will therefore succeed, although the code in the integrated system may fail. If your intent is that the code works on different systems, you might even encounter the situation that the code works in one integration (maybe Linux) but fails in another (maybe Windows).
Therefore, for code like yours, unit-testing and thus mocking for unit-testing does not have much value.
Monkey patch the handler and send the signal when testing?
import os
import signal
import sys
import time
# your handler
def sigterm_handler(signum, frame):
print("Handled")
pid = os.getpid() # type: int FIXME: what's this for?
sys.exit(0)
signal.signal(signal.SIGTERM, sigterm_handler)
# Mock out the existing sigterm_handler
_handled = False
def mocked_sigterm_handler(signum, frame):
print("Mocked")
_handled = True
# register the handler
signal.signal(signal.SIGTERM, mocked_sigterm_handler)
# test sending the signal
os.kill(os.getpid(), signal.SIGTERM)
print(f"done ({_handled})")
# reset your handler?
signal.signal(signal.SIGTERM, sigterm_handler)
If you want to test you handler itself you'll probably have to put some kind of code like this.. in the handler which is not beautiful.
if _unittesting_sigterm_handler:
_handled = True
else:
sys.exit(0)
and then you can just call the handler directly (or pass the test flag in the call).
_unittesting_sigterm_handler = True
sigterm_handler(0, None)
With pytest is there a way to run cleanup code on a specific test function/method alone. I know we can do this to run for each test function. But here I want to place some cleanup logic specific to a single test function.
I can just put cleanup code at the end of the test. But if test fails then cleanup wont be done.
Create a fixture with your cleanup code and inject it only into the one test by using the fixture as an argument for your test or by explicitly marking the test with the pytest.mark.usefixtures decorator.
import pytest
#pytest.fixture
def my_cleanup_fixture():
# Startup code
...
yield
# Cleanup code
...
#pytest.mark.usefixtures('my_cleanup_fixture')
def test_with_special_cleanup():
pass
my_cleanup_fixture has scope function by default, so the startup and cleanup code will run for each function it is injected.
I have some celery task. I want to test it via unittest.
I'm doing something very similar to:
class TestMe(unittest.TestCase):
def test_celery_task(self):
self.assertRaises(ValueError, celery_task.apply, args)
what is strange for me:
this assert fails, because ValueError not raised, but during executing process I can see ValueError as a result of this celery task.
I'm not sure, but it looks like assert is checking faster than ValueError is rising.
Is it possible to check the result of executed celery task?
or how it may be tested?
That can't possibly work. When you enqueue a Celery task, all that happens is that you put a message into the queue for a separate process to pick up; it is that process that runs the task and, potentially, raises the exception.
If you want to check that the task itself raises ValueError, then you should call the task, not the delay function:
self.assertRaises(ValueError, celery_task, args)
I see 3 options here.
1) Try to call get() on apply(). Here is what you will get:
class TestMe(unittest.TestCase):
def test_celery_task(self):
self.assertRaises(ValueError, celery_task.apply().get(), args)
2) You can either enable eager mode by setting 'task_always_eager' to True, however it does not guarantee that your code will be able to catch up.
3) A better option would be to mock the celery tasks. From the point of unit testing it is not actually correct to test a unit of code with actual 'alive' part of systems like celery.
Here is a sample of code taken from celery testing documentation.
from pytest import raises
from celery.exceptions import Retry
# for python 2: use mock.patch from `pip install mock`.
from unittest.mock import patch
from proj.models import Product
from proj.tasks import send_order
class test_send_order:
#patch('proj.tasks.Product.order') # < patching Product in module above
def test_success(self, product_order):
product = Product.objects.create(
name='Foo',
)
send_order(product.pk, 3, Decimal(30.3))
product_order.assert_called_with(3, Decimal(30.3))
#patch('proj.tasks.Product.order')
#patch('proj.tasks.send_order.retry')
def test_failure(send_order_retry, product_order):
product = Product.objects.create(
name='Foo',
)
# set a side effect on the patched method
# so that it raises the error we want.
product_order.side_effect = OperationalError()
with raises(Retry):
send_order(product.pk, 3, Decimal(30.6))
So far I've used nosetests with just one process and everything works fine.
To ensure my setUp is only executed once, I'm using a boolean var.
def setUp(self):
if not self.setupOk:
selTest.setupOk = True
# start selenium
# do other stuff which will be needed for all other tests to be able to run
Now I would like to run nosetests with the option --processes=5
How can I ensure that setUp(self) is only execued by one process (while the other processes are waiting).
I've tried to work with
def setUp(self):
lock = multiprocessing.Lock()
lock.acquire()
if not self.setupOk:
selTest.setupOk = True
# start selenium
# do other stuff which will be needed for all other tests to be able to run
lock.release()
but this doesn't seems to work.
setUp will be called before every test is run. If you want a method to execute just once, you can use setUpClass:
#classmethod
def setUpClass(cls):
print "do stuff which needs to be run once"
Similar to this question, I'd like to have Nose run a test (or all tests) n times -- but not in parallel.
I have a few hundred tests in a project; some are some simple unit tests. Others are integration tests w/ some degree of concurrency. Frequently when debugging tests I want to "hit" a test harder; a bash loop works, but makes for a lot of cluttered output -- no more nice single "." for each passing test. Having the ability to beat on the selected tests for some number of trials seems like a natural thing to ask Nose to do, but I haven't found it anywhere in the docs.
What's the simplest way to get Nose to do this (other than a bash loop)?
You can write a nose test as a generator, and nose will then run each function
yielded:
def check_something(arg):
# some test ...
def test_something():
for arg in some_sequence:
yield (check_something, arg)
Using nose-testconfig, you could make the number of test runs a command line argument:
from testconfig import config
# ...
def test_something():
for n in range(int(config.get("runs", 1))):
yield (check_something, arg)
Which you'd call from the command line with e.g.
$ nosetests --tc=runs:5
... for more than one run.
Alternatively (but also using nose-testconfig), you could write a decorator:
from functools import wraps
from testconfig import config
def multi(fn):
#wraps(fn)
def wrapper():
for n in range(int(config.get("runs", 1))):
fn()
return wrapper
#multi
def test_something():
# some test ...
And then, if you want to divide your tests into different groups, each with their own command line argument for the number of runs:
from functools import wraps
from testconfig import config
def multi(cmd_line_arg):
def wrap(fn):
#wraps(fn)
def wrapper():
for n in range(int(config.get(cmd_line_arg, 1))):
fn()
return wrapper
return wrap
#multi("foo")
def test_something():
# some test ...
#multi("bar")
def test_something_else():
# some test ...
Which you can call like this:
$ nosetests --tc=foo:3 --tc=bar:7
You'll have to write a script to do this, but you can repeat the test names on the commandline X times.
nosetests testname testname testname testname testname testname testname
etc.
Solution I ended up using is create sh script run_test.sh:
var=0
while $1; do
((var++))
echo "*** RETRY $var"
done
Usage:
./run_test.sh "nosetests TestName"
It runs test infinitely but stops on first error.
One way is in the test itself:
Change this:
class MyTest(unittest.TestCase):
def test_once(self):
...
To this:
class MyTest(unittest.TestCase):
def assert_once(self):
...
def test_many(self):
for _ in range(5):
self.assert_once()
There should never be a reason to run a test more than once. It's important that your tests are deterministic (i.e. given the same state of the codebase, they always produce the same result.) If this isn't the case, then instead of running tests more than once, you should redesign the tests and/or code so that they are.
For example, one reason why tests fail intermittently is a race condition between the test and the code-under-test (CUT). In this circumstance, a naive response is to add a big 'voodoo sleep' to the test, to 'make sure' that the CUT is finished before the test starts asserting.
This is error-prone though, because if your CUT is slow for any reason (underpowered hardware, loaded box, busy database, etc) then it will fail sporadically. A better solution in this instance is to have your test wait for an event, rather than sleeping.
The event could be anything of your choosing. Sometimes, events you can use are already being generated (e.g. Javascript DOM events, the 'pageRendered' kind of events that Selenium tests can make use of.) Other times, it might be appropriate for you to add code to your CUT which raises an event when it's done (perhaps your architecture involves other components that are interested in events like this.)
Often though, you'll need to re-write the test such that it tries to detect whether your CUT is finished executing (e.g. does the output file exist yet?), and if not, sleeps for 50ms and then tries again. Eventually it will time out and fail, but only do this after a very long time (e.g. 100 times the expected execution time of your CUT)
Another approach is to design your CUT using 'onion/hexagonal/ports'n'adaptors' principles, which insists your business logic should be free of all external dependencies. This means that your business logic can be tested using plain old sub-millisecond unit tests, which never touch the network or filesystem. Once this is done, you need far fewer end-to-end system tests, because they are now serving just as integration tests, and don't need to try to manipulate every detail and edge-case of your business logic going through the UI. This approach will also yield big benefits in other areas, such as improved CUT design (reducing dependencies between components), tests are much easier to write, and the time taken to run the whole test suite is much reduced.
Using approaches like the above can entirely eliminate the problem of unreliable tests, and I'd recommend doing so, to improve not just your tests, but also your codebase, and your design abilities.