I'm having trouble understanding, in pytest, when (or why) i need to add the #pytest.hookimpl(hookwrapper=True) tag to all my pytest_runtest_* calls.
basically, in conftest, i have implemented all of those calls to do do something before yeilding, and then doing something after, such as:
def pytest_runtest_call(item: Any) -> None:
""" TODO: method docstring """
item.session.test_info["on_case_number"] += 1
pytest_log = getLogger("runtest")
pytest_log.debug(f'= runtest started [{count_str(item.session.test_info)}]')
outcome = yield
setattr(item, "case_markers", item.own_markers)
module_name, class_name, case_name = item.nodeid.split("::")
trunk = item.session.test_info["items"][module_name]["classes"][class_name]
try:
# set a report attribute for each phase of a call, which can
# be "setup", "call", "teardown"
outcome.get_result()
except KeyboardInterrupt:
raise KeyboardInterrupt("Caught Keyboard Termination")
except Exception as e:
trunk["cases"][case_name]["passed"] = False
_process_exception(e, item, pytest_log)
pytest_log.debug(f'= runtest completed [{count_str(item.session.test_info)}]')
When I run a test case, this never gets called. However, if I prepend to tag
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item: Any) -> None:
...
It does get called.
But I was under the (probalby false) impression that a certain set of methods within pytest get called automagically if they are defined, including the pytest_runtest_* set (and in fact, pytest_collection_modifyitems for instance does get called eventually w/o the tag).
I'm confused as to why this is the case. Or if I am just not getting/doing something correctly.
Related
I want to clean up some files after all tests pass. If they fail, keep them for debug. I read https://docs.pytest.org/en/latest/example/simple.html#making-test-result-information-available-in-fixtures so I have the following in my conftest.py:
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
# execute all other hooks to obtain the report object
outcome = yield
rep = outcome.get_result()
# set a report attribute for each phase of a call, which can
# be "setup", "call", "teardown"
setattr(item, "rep_" + rep.when, rep)
#pytest.fixture(scope="module", autouse=True)
def teardown(request):
yield
# request.node is an "item" because we use the default
# "function" scope
if request.node.rep_setup.failed:
print("setting up a test failed!", request.node.nodeid)
elif request.node.rep_setup.passed:
#clean up my files
however, I got the error:
AttributeError: 'Module' object has no attribute 'rep_setup'
The only difference from doc example is that my teardown has 'scope=module'. But I have to do this because I want to clean up files after all tests pass, some files are used by all tests. If I use the default scope which is 'function' level, it will clean up after each test case rather than after the whole module. How can I fix this?
Update: Before I had 'hook', I still had the teardown which is "module" level, and it worked fine, meaning it cleaned up all files for me after all tests running, the only problem is that it will clean up for me no matter tests pass or fail.
If you are in module scope, request.node represents the module, not a single test. If you want just check for failed tests, you can check the session:
#pytest.fixture(scope="module", autouse=True)
def teardown(request):
yield
if request.session.testsfailed > 0:
print(f"{} test(s) failed!", request.session.testsfailed)
else:
# clean up my files
I'm not sure if there is any information about setup failures in the request at this point, if you are only interested in these.
In this case you could implement a file scoped fixture which sets a flag in case of a setup failure, and use that, something like:
SETUP_FAILED = False
#pytest.fixture(autouse=True)
def teardown_test(request):
yield
if request.node.rep_setup.failed:
global SETUP_FAILED
SETUP_FAILED = True
#pytest.fixture(scope="module", autouse=True)
def teardown_module():
global SETUP_FAILED
SETUP_FAILED = False
yield
if SETUP_FAILED:
print("At least one test setup failed!")
else:
# clean up my files
This is not nice, and maybe someone knows a better solution, but it will work.
You could also collect information about the tests where the setup failed if needed.
I have a function in a python program which does a function call twice:
def add_user(uname,dserver,pwd,dinstance,proc1, query1):
db_conn = db_connect(uname,dserver,pwd,dinstance)
if db_conn is not_conn:
print("Failed to connect")
sys.exit(-1)
db_conn.run_proc(proc1)
if db_conn.error_msg:
print("Failed procedure")
sys.exit(-1)
db_conn.run_query(query1)
if db_conn.err_msg:
print("Failed query")
sys.exit(-1)
Now the unit test is as follows:
#patch('mydir.proj_dir.db_execu.db_connect')
def test_add_user(self, mock_conn):
mock_conn.return_value.not_conn.return_value = True
mock_conn.return_value.run_proc.return_value = True
mock_conn.return_value.run_query.return_value = True
mock_conn.return_value.err_msg.side_effect= [True, False]
with self.assertRaises(SystemExit):
add_user(name,dserver,pwd,dinstance,proc1, query1)
print("failed query")
My objective is to test the second error condition. But after adding side_effect it only going to first condition which is displaying as "Failed procedure". I want to test "Failed Query" condition. First two error conditions I have tested but the third one is failing and calling the second condition always. Please advise.
It's not super clear what you're asking exactly. You're calling add_user only once so you're only testing a single "path", you need to call it multiple times to test multiple "paths" through it.
That aside, there's a huge issue with your understanding of mock return_value and side_effect configure mocks as callables, when accessing attributes but not calling them, you're just getting the next mock in the chain, you're not using any of your configuration. mock also supports configuring some of the "magic methods" of the data model, but it does not support configuring __getattr__ (the "simple" attribute access) as that's used internally by mock.
As a result, you should only use mock to patch / replace db_connect to return a pseudo-connection, but rather than a mock that pseudo-connection should be a fake: an object you build yourself which looks like a connection but behaves however you want it e.g.
class PseudoConnection:
def __init__(self, fail_proc=False, fail_query=False):
self._fail_proc = fail_proc
self._fail_query = fail_query
self.err_msg = None # or error_msg, or both
def run_proc(self, proc):
self.err_msg = self._fail_proc
def run_query(self, query):
self.err_msg = self._fail_query
Then you just configure mock_conn.return_value = PseudoConnection(True, False) to test the first case and mock_conn.return_value = PseudoConnection(False, True) for the second.
Also as hinted by the comment you're accessing both error_msg and err_msg, and only attempting to configure err_msg. I don't know which is right, but I doubt both are.
I'm using Python's unittest with pytest for integration testing a library against a third-party API.
Some of the API calls are temporarily returning an error which raises a specific exception in my code. This behaviour is fine in the code.
However, rather than having the tests fail, I'd rather skip these temporary errors.
I have over 150 tests. Rather than rewriting each and every test like this:
class TestMyLibrary(unittest.TestCase):
def test_some_test(self):
try:
// run the test as normal
// assert the normal behaviour
except SomeException:
// skip the test
def test_some_other_test(self):
try:
// run the test as normal
// assert the normal behaviour
except SomeException:
// skip the test
Can I rather wrap them all somehow at the class level, or similar?
If you expect this exception why don't you check its raised when it should?
You can use :
pytest.raises(Exceptiontype, Foo())
This can be done with a decorator. For example:
def handle_lastfm_exceptions(f):
def wrapper(*args, **kw):
try:
return f(*args, **kw)
except pylast.WSError as e:
if (str(e) == "Invalid Method - "
"No method with that name in this package"):
msg = "Ignore broken Last.fm API: " + str(e)
print(msg)
pytest.skip(msg)
else:
raise(e)
return wrapper
And then decorate the problematic functions:
class TestMyLibrary(unittest.TestCase):
#handle_lastfm_exceptions
def test_some_bad_test(self):
// run the test as normal
// assert the normal behaviour
def test_some_good_test(self):
// run the test as normal
// assert the normal behaviour
had the same problem (instable 3rd party library, waiting for fix...). ended up with something like this:
def pytest_runtest_makereport(item, call):
from _pytest.runner import pytest_runtest_makereport as orig_pytest_runtest_makereport
tr = orig_pytest_runtest_makereport(item, call)
if call.excinfo is not None:
if call.excinfo.type == SomeExceptionFromLibrary:
tr.outcome = 'skipped'
tr.wasxfail = "reason: SomeExceptionFromLibrary. shame on them..."
return tr
works like a charm
I'm trying to add SimPy simulation to a project I'm working on and I have some confusion about version 3's release/request.
I was able to implement resources using a 'with'-block without trouble but in my situation I want to request/release a resource without using a 'with'-block.
However, I cannot find an example of this using SimPy 3. I read the documentation/source regarding resources but still can't get it quite right. Could someone explain how to properly:
...
Request a Resource with the method: 'request()'
...
Release that Resource with the method: 'release()'
...
Thanks, and sorry for the bother.
PS: I'm intending to use Resources.resource
If you want to use a resource without a with block (and you know you won’t get interrupted), its just:
req = resource.request()
yield req
# do stuff
resource.release(req)
Using with on an object calls __enter__ when you enter the with block, and __exit__ when you leave. So when you do
res = resource.Resource()
with res.request() as req:
# stuff
You're really calling __enter__ on a Request object, doing #stuff then calling __exit__:
class Request(base.Put):
def __exit__(self, exc_type, value, traceback):
super(Request, self).__exit__(exc_type, value, traceback)
self.resource.release(self)
class Put(Event): # base.Put
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
# If the request has been interrupted, remove it from the queue:
if not self.triggered:
self.resource.put_queue.remove(self)
So, the with block is equivalent to this:
res = resource.Resource(...)
req = res.request()
#stuff
if not req.triggered:
res.put_queue.remove(req)
res.release(req)
However, the with block is also making sure that the cleanup code is called no matter what exceptions are thrown during #stuff. You'll lose that with the above code.
It's all outlined in PEP343;
with EXPR as VAR:
BLOCK
becomes:
mgr = (EXPR)
exit = type(mgr).__exit__ # Not calling it yet
value = type(mgr).__enter__(mgr)
exc = True
try:
try:
VAR = value # Only if "as VAR" is present
BLOCK
except:
# The exceptional case is handled here
exc = False
if not exit(mgr, *sys.exc_info()):
raise
# The exception is swallowed if exit() returns true
finally:
# The normal and non-local-goto cases are handled here
if exc:
exit(mgr, None, None, None)
This is exactly how python uses with... as... blocks, but I'm presuming there's some reason you don't want to use these. If that's the case, then you just need the __enter__ and __exit__ functions. The way i think of it, is __enter__ sets everything up, and __exit__ does all the cleanup.
I'm using the following code in my testing framework:
testModules = ["test_foo", "test_bar"]
suite = unittest.TestLoader().loadTestsFromNames(testModules)
runner = unittest.TextTestRunner(sys.stdout, verbosity=2)
results = runner.run(suite)
return results.wasSuccessful()
Is there a way to make the reporting (runner.run?) abort after the first failure to prevent excessive verbosity?
Nine years after the question was asked, this is still one of the top search results for "python unit test fail early" and, as I discovered when looking at the other search results, these answers are no longer correct for more recent versions of the unittest module.
The documentation for the unittest module https://docs.python.org/3/library/unittest.html#command-line-options and https://docs.python.org/2.7/library/unittest.html#command-line-options show that there is an argument, failfast=True, that can be added to unittest.main, or equivalently a command line option, -f, or --failfast, to stop the test run on the first error or failure. This option was added in version 2.7. Using that option is a lot easier than the previously-necessary workarounds suggested in the other answers.
That is, simply change your
unittest.main()
to
unittest.main(failfast=True)
It's a feature. If you want to override this, you'll need to subclass TestCase and/or TestSuite classes and override logic in the run() method.
P.S.:
I think you have to subclass unittest.TestCase and override method run() in your class:
def run(self, result=None):
if result is None: result = self.defaultTestResult()
result.startTest(self)
testMethod = getattr(self, self._testMethodName)
try:
try:
self.setUp()
except KeyboardInterrupt:
raise
except:
result.addError(self, self._exc_info())
return
ok = False
try:
testMethod()
ok = True
except self.failureException:
result.addFailure(self, self._exc_info())
result.stop()
except KeyboardInterrupt:
raise
except:
result.addError(self, self._exc_info())
result.stop()
try:
self.tearDown()
except KeyboardInterrupt:
raise
except:
result.addError(self, self._exc_info())
ok = False
if ok: result.addSuccess(self)
finally:
result.stopTest(self)
(I've added two result.stop() calls to the default run definition).
Then you'll have to modify all your testcases to make them subclasses of this new class, instead of unittest.TestCase.
WARNING: I didn't test this code. :)
Based on Eugene's guidance, I've come up with the following:
class TestCase(unittest.TestCase):
def run(self, result=None):
if result.failures or result.errors:
print "aborted"
else:
super(TestCase, self).run(result)
While this works fairly well, it's a bit annoying that each individual test module has to define whether it wants to use this custom class or the default one (a command-line switch, similar to py.test's --exitfirst, would be ideal)...
Building on AnC's answer, this is what I'm using...
def aborting_run(self, result=None):
if result.failures or result.errors:
print "aborted"
else:
original_run(self, result)
original_run = unittest.TestCase.run
unittest.TestCase.run = aborting_run