I'm trying to write a Ruby script to execute all Python tests in a directory.
Here is the main part of the script:
system("python #{solution}/tests.py")
Is there a way to somehow understand if the tests have passed and to show output only for the ones that failed (or whatever else)? I just want to be able to tell for each test if it passed or not. (If not possible with Ruby, I'm open for solutions with Python, too)
The tests all look like this one:
import unittest
from solution import my_function
class MyFunctionTest(unittest.TestCase):
def test_my_function(self):
self.assertEqual(42, my_function(42))
if __name__ == '__main__':
unittest.main()
Thanks very much in advance! :)
You can print the output for failing tests with nosetests, which runs them all and condenses the passing tests:
$ nosetests
.............
----------------------------------------------------------------------
Ran 13 tests in 0.015s
OK
Adding a failing test:
$ nosetests
F.............
======================================================================
FAIL: test_failed (x32.tests.failing_test.FailingTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/leigh/Projects/x32/x32/tests/failing_test.py", line 5, in test_failed
self.assertEqual(1, 2)
AssertionError: 1 != 2
----------------------------------------------------------------------
Ran 14 tests in 0.013s
FAILED (failures=1)
There's a guide for nose here, as well: https://nose.readthedocs.org/en/latest/
Alternatively you can go to the parent directory and run something like python -m pytest . to execute the tests, however this will tell you which ones pass as well.
Related
I'd like to run a single test contained in a subclass of unittest.TestCase using nose2 following How to run specific test in Nose2, but it doesn't seem to work for me. I'm using the following example script, which I've named mickey_mouse_test.py:
import unittest
class TestMickeyMouse(unittest.TestCase):
def test_1plus1is2(self):
self.assertTrue(1+1 == 2)
def test_to_uppercase(self):
self.assertEqual("hello".upper(), "HELLO")
if __name__ == "__main__":
unittest.main()
If I run nose2 mickey_mouse_test in the same directory, it runs all the tests in the module:
kurt#kurt-ThinkPad:~/Documents/Scratch$ nose2 mickey_mouse_test
..
----------------------------------------------------------------------
Ran 2 tests in 0.001s
OK
However, if I try to run just test_to_uppercase like so I get an error:
kurt#kurt-ThinkPad:~/Documents/Scratch$ nose2 mickey_mouse_test.test_to_uppercase
E
======================================================================
ERROR: mickey_mouse_test.test_to_uppercase (nose2.loader.LoadTestsFailure)
----------------------------------------------------------------------
AttributeError: module 'mickey_mouse_test' has no attribute 'test_to_uppercase'
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (errors=1)
If I use the -s option I still get an error, albeit a different one:
kurt#kurt-ThinkPad:~/Documents/Scratch$ nose2 -s mickey_mouse_test.test_to_uppercase
E
======================================================================
ERROR: mickey_mouse_test.test_to_uppercase (nose2.loader.LoadTestsFailure)
----------------------------------------------------------------------
OSError: /home/kurt/Documents/Scratch/mickey_mouse_test.test_to_uppercase is not a directory
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (errors=1)
I've also tried reading the "Specifying tests to run" section in http://nose2.readthedocs.io/en/latest/usage.html, in which it is stated that the 'Python object part' should be a 'dotted name'. I don't see why in this case, mickey_mouse_test.test_to_uppercase is not a 'dotted name'. Any ideas why this is not working?
Here is a way to run just test_to_uppercase without using nose2 (following Running single test from unittest.TestCase via command line):
kurt#kurt-ThinkPad:~/Documents/Scratch$ python mickey_mouse_test.py TestMickeyMouse.test_to_uppercase
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
I am having some trouble trying to debug some unit tests through the pudb debugger.
The tests run fine with python, but I had no luck runnign them with pudb.
I isolated the problem, getting to the following sample code:
class Math:
def pow(self, x, y):
return x ** y
import unittest
class MathTest(unittest.TestCase):
def testPow23(self):
self.assertEquals(8, Math().pow(2, 3))
def testPow24(self):
self.assertEquals(16, Math().pow(2, 4))
if __name__ == '__main__':
unittest.main()
The tests run fine:
$ python amodule.py
.
----------------------------------------------------------------------
Ran 2 tests in 0.001s
OK
But if running through pudb, it gives me the output:
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
I've tried running using pudb amodule.py and also with python -m pudb.run amodule.py, but it makes no difference -- no tests are run in one or another way.
Should I be doing something different to debug unit tests using pudb?
Try placing a breakpoint on a useful line in your code:
from pudb import set_trace; set_trace()
The ways you tried to launch it might interfere with test discovery and/or not run your script with a __name__ of '__main__'.
Since this is a popular question, I feel I should also mention that most test running tools will require you to pass in a switch to prevent it from capturing the standard output and input (usually it's -s).
So, remember to run pytest -s when using Pytest, or nosetests -s for Nose, python manage.py test -s for Django tests, or check the documentation for your test running tool.
You can set a breakpoint even easier by:
import pudb; pu.db
Consider a unittest suite like this:
import unittest
class TestClass(unittest.TestCase):
def test_set_trace(self):
raise Exception
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(TestClass)
unittest.TextTestRunner(verbosity=2).run(suite)
When I put the following in the script and run it like so: python -m pdb tmp.py I would expect the Python debugger to catch the exception and bring me inside the test_set_trace method, however unittest catches the exception and hides it from pdb like so:
$ python -m pdb tmp.py
> tmp.py(1)<module>()
-> import unittest
(Pdb) c
test_set_trace (__main__.TestClass) ... ERROR
======================================================================
ERROR: test_set_trace (__main__.TestClass)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tmp.py", line 5, in test_set_trace
raise Exception
Exception
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (errors=1)
The program finished and will be restarted
> tmp.py(1)<module>()
-> import unittest
(Pdb)
This is annoying when writing my tests as I need to go into the testing code and specify pdb.set_trace() manually, then step to the frame that contains the actual bug. Does anyone know of a workaround? Putting an explicit import pdb inside the testing code doesn't help, even if I put the import statement within the actual offending method.
Use TestSuite.debug.
debug()
Run the tests associated with this suite without collecting
the result. This allows exceptions raised by the test to be propagated
to the caller and can be used to support running tests under a
debugger.
Replace following line:
unittest.TextTestRunner(verbosity=2).run(suite)
with:
suite.debug()
I'm running nosetests which have a setup function that needs to load a different database than the production database. The ORM I'm using is peewee which requires that the database for a model is set in the definition.
So I need to set a conditional variable but I don't know what condition to use in order to check if nosetest is running the file.
I read on Stack Overflow that you can check for nose in sys.modules but I was wondering if there is a more exact way to check if nose is running.
Perhaps examining sys.argv[0] to see what command is running?
Examining sys.argv might work, but you can execute nose either with nosetests or python -m nose, which obviously will give you a different result.
I think the more robust way is to inspect the stack and see if the code is being called through a package called nose.
Example code:
import inspect
import unittest
def is_called_by_nose():
stack = inspect.stack()
return any(x[0].f_globals['__name__'].startswith('nose.') for x in stack)
class TestFoo(unittest.TestCase):
def test_foo(self):
self.assertTrue(is_called_by_nose())
Example usage:
$ python -m nose test_caller
.
----------------------------------------------------------------------
Ran 1 test in 0.009s
OK
$ nosetests test_caller
.
----------------------------------------------------------------------
Ran 1 test in 0.009s
OK
$ python -m unittest test_caller
F
======================================================================
FAIL: test_foo (test_caller.TestFoo)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_caller.py", line 14, in test_foo
self.assertTrue(is_called_by_nose())
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 0.004s
FAILED (failures=1)
No matter what I do sys.exit() is called by unittest, even the most trivial examples. I can't tell if my install is messed up or what is going on.
IDLE 1.2.2 ==== No Subprocess ====
>>> import unittest
>>>
>>> class Test(unittest.TestCase):
def testA(self):
a = 1
self.assertEqual(a,1)
>>> unittest.main()
option -n not recognized
Usage: idle.pyw [options] [test] [...]
Options:
-h, --help Show this message
-v, --verbose Verbose output
-q, --quiet Minimal output
Examples:
idle.pyw - run default set of tests
idle.pyw MyTestSuite - run suite 'MyTestSuite'
idle.pyw MyTestCase.testSomething - run MyTestCase.testSomething
idle.pyw MyTestCase - run all 'test*' test methods
in MyTestCase
Traceback (most recent call last):
File "<pyshell#7>", line 1, in <module>
unittest.main()
File "E:\Python25\lib\unittest.py", line 767, in __init__
self.parseArgs(argv)
File "E:\Python25\lib\unittest.py", line 796, in parseArgs
self.usageExit(msg)
File "E:\Python25\lib\unittest.py", line 773, in usageExit
sys.exit(2)
SystemExit: 2
>>>
In new Python 2.7 release, unittest.main() has a new argument.
If 'exit' is set to False, sys.exit() is not called during the execution of unittest.main().
Your example is exiting on my install too. I can make it execute the tests and stay within Python by changing
unittest.main()
to
unittest.TextTestRunner().run(unittest.TestLoader().loadTestsFromTestCase(Test))
More information is available here in the Python Library Reference.
Don't try to run unittest.main() from IDLE. It's trying to access sys.argv, and it's getting the args that IDLE was started with. Either run your tests in a different way from IDLE, or call unittest.main() in its own Python process.
Pop open the source code to unittest.py. unittest.main() is hard-coded to call sys.exit() after running all tests. Use TextTestRunner to run test suites from the prompt.
It's nice to be able to demonstrate that your tests work when first trying out the unittest module, and to know that you won't exit your Python shell. However, these solutions are version dependent.
Python 2.6
I'm using Python 2.6 at work, importing unittest2 as unittest (which is the unittest module supposedly found in Python 2.7).
The unittest.main(exit=False) doesn't work in Python 2.6's unittest2, while JoeSkora's solution does, and to reiterate it:
unittest.TextTestRunner().run(unittest.TestLoader().loadTestsFromTestCase(Test))
To break this down into its components and default arguments, with correct semantic names for the various composed objects:
import sys # sys.stderr is used in below default args
test_loader = unittest.TestLoader()
loaded_test_suite = test_loader.loadTestsFromTestCase(Test)
# Default args:
text_test_runner = unittest.TextTestRunner(stream=sys.stderr,
descriptions=True,
verbosity=1)
text_test_runner.run(loaded_test_suite)
Python 2.7 and 3
In Python 2.7 and higher, the following should work.
unittest.main(exit=False)
try:
sys.exit()
except SystemExit:
print('Simple as that, but you should really use a TestRunner instead')