I wrote a python script to do all my tests automatically for me, and generate a HTML report. I discovered discover for unittests the other day which lets me run all the unittests in a given directory without explicitly naming them, and I'd really like to be able to do my doctests the same way, rather than having to import each module explicitly.
I found some info on how to do this at https://docs.python.org/2/library/doctest.html but didn't really get it. Could you please help me with using discover with my doctests?
Python test discovery with doctests, coverage and parallelism is related, but still doesn't answer my question.
coverage_module
import coverage
import doctest
import unittest
import os
# import test_module
import my_module
cov = coverage.Coverage()
cov.start()
# running doctest by explicity naming the module
doctest.testmod(my_module)
# running unittests by just specifying the folder to look into
testLoad = unittest.TestLoader()
testSuite = testLoad.discover(start_dir=os.getcwd())
runner = unittest.TextTestRunner()
runner.run(testSuite)
cov.stop()
cov.save()
cov.html_report()
print "tests completed"
test_module
import unittest
import doctest
from my_module import My_Class
class My_Class_Tests(unittest.TestCase):
def setUp(self):
# setup variables
def test_1(self):
# test code
# The bit that should load up the doctests? What's loader, tests, and ignore though?
# Is this in the right place?
def load_tests(loader, tests, ignore):
tests.addTests(doctest.DocTestSuite(module_with_doctests))
return tests
if __name__ == '__main__':
unittest.main()
Lets figure out what's happening there
1) unittest.discovery
It has no clue of doctests as doctests is a different framework.
So unittest isn't supposed to discover doctests out of the box.
That means you'll need to glue them together by hand
2) doctest
It's essentially a separate framework although it has some glueing classes to convert doctests into unittest-like TestCases.
https://docs.python.org/2.7/library/doctest.html#doctest.DocTestSuite
3) discover
Didn't get what discover you mean, I suppose it's
python -m unittest discover
If not and you're talking about https://pypi.python.org/pypi/discover then just forget about it - it's a backport for earlier versions of python
4) what to do
either scatter a lot of load_tests hooks across your code as described here https://docs.python.org/2.7/library/doctest.html#unittest-api or code a method to collect all the modules your have in one place and convert them into a DocTestSuite[s] https://docs.python.org/2.7/library/doctest.html#doctest.DocTestSuite
But honestly neither approach makes any sense nowadays as it boils down to:
$ py.test --doctest-modules
or
$ nosetests --with-doctest
Of course coverage and lots of bells & whistles are also supplied by these frameworks and you may keep sticking to unittest.TestCase, and you won't even need to create a coverage_module so I would dig into one of them rather then trying to come up with your own solution
Related
after watching a couple of presentations about django testing, I want to code my own TestRunner in order to skip django tests, and create better packages structures for my tests.
The problem is that we've changed the project structure and the test runner can't find the right path to do the tests discovery. This is how my project looks like:
project/
-src/
- project_name/
- apps/
- test/ # Not a good name, i know, will change it
- some_app/
- test_models.py
- manage.py
- development.db
Now, in order to test test_models.py I want to do this:
$ cd project/src/
$ python manage.py test some_app.test_models
The problem is that the test runner can't find that package (some_app) and module (test_models.py). It changes if I hardcode the name in the test runner, but i don't like to do it. Here's what I do to make it work.
test_labels = ["%s.%s" % ("project_name.test", l)
for l in test_labels
if not l.startswith("project_name.test")]
So, if you do
$ python manage.py test some_app.test_models
It will be rewritten to:
$ python manage.py test project_name.test.some_app.test_models
And that works fine.
I tried doing sys.path.append("(...)/project_name/test) but doesn't work neither.
This is the code of my TestRunner:
class DiscoveryDjangoTestSuiteRunner(DjangoTestSuiteRunner):
"""A test suite runner that uses unittest2 test discovery.
It's better than the default django test runner, becouse it
doesn't run Django tests and let you put your tests in different
packages, modules and classes.
To test everything in there:
$ ./manage.py test
To test a single package/module:
$ ./manage.py test package
$ ./manage.py test package.module
To test a single class:
$ ./manage.py test package.module.ClassName
"""
def build_suite(self, test_labels, extra_tests=None, **kwargs):
suite = None
discovery_root = settings.TEST_DISCOVERY_ROOT
if test_labels:
# This is where I append the path
suite = defaultTestLoader.loadTestsFromNames(test_labels)
# if single named module has no tests, do discovery within it
if not suite.countTestCases() and len(test_labels) == 1:
suite = None
discovery_root = import_module(test_labels[0]).__path__[0]
if suite is None:
suite = defaultTestLoader.discover(
discovery_root,
top_level_dir=settings.BASE_PATH,
)
if extra_tests:
for test in extra_tests:
suite.addTest(test)
return reorder_suite(suite, (TestCase,))
Your Python import hierarchy is rooted at project/src. Thus, the correct Python import path for your test_models module is project_name.test.some_app.test_models, so that's what I would expect to pass in as a test label.
But you don't like typing the project_name.test prefix every time you want to run a specific test module, since all your tests will be located there. That's fine: you're choosing to introduce some implicit non-obvious behavior in exchange for some convenience. You definitely should not add anything to sys.path in order to achieve this: the key to Python import sanity is having your import hierarchy for a given codebase rooted in one and exactly one place; overlapping sys.path entries will cause problems like doubled imports of the same module under different names.
Really all you want is a UI convenience, and it looks to me like the test-label-munging code you show is the obvious way to implement that convenience. You don't like having the project_name.test prefix hardcoded, but it's going to have to be hardcoded somewhere: there's no way the test runner is going to magically figure out that you want to prepend test labels with project_name.test. If you want your TestRunner to be more generic, you can pull it out into a setting like BASE_TEST_MODULE or some such and prepend the value of that setting to each test label.
Before you continue investing more time into your custom TestRunner, I would definitely recommend that you take a look at django-nose.
The custom test runner provided by django-nose implements nose's test runner which is extremely flexible and provides a lot of options for running your tests. It seamlessly overrides the default test management command and allows you to configure default test options in your project's settings module.
I'm really recommending it for several reasons:
The options for the test command are fully documented (take look at the output)
nose provides a lot of approaches for test discovery
Chances are your colleagues are already seasoned nose users
You didn't have to write the TestRunner class yourself
I'm struggling to know where to start with unittest, having read the dive-into-python tutorial and looked at http://pyunit.sourceforge.net/.
I've got a piece of analysis software (call it 'prog.exe') which uses python for its input decks. I've started writing a python module which I'm going to import from that input deck to provide some useful functionality. So, running one of these analyses will go like this:
prog.exe inputdeck.py
where inputdeck.py contains:
from mymodule import mystuff
So how do I set up and run tests on mymodule? Should the above be in a system call in a setUp method of the test, or what?
Ok - solution:
Don't use unittest.main() as that's the command line tool. Instead call the appropriate unittest methods directly as follows:
From the command line run:
prog.exe mytests.py
where mytests.py contains:
import unittest
# ... code to run the analysis which we'll use for the tests ...
# ... test definitions ...
suite = unittest.TestLoader().loadTestsFromTestCase(test_cases)
unittest.TextTestRunner().run(suite)
See example at http://docs.python.org/release/2.6.7/library/unittest.html#unittest.TextTestRunner
Pyunit is a little bit outdated (2001), it is now completely included in python core distribution (http://docs.python.org/library/unittest.html). You should start read this documentation, especially the basic example part.
To test your module you'll have to create a file, let's call it mymodule_test.py and put in it something like this :
import unittest
from mymodule import mystuff
class MyTestCase(unittest.TestCase):
def test_01a(self):
""" test mystuff"""
self.failUnless(mystuff.do_the_right_stuff())
if __name__ == '__main__':
unittest.main()
and run it with python mymodule_test.py
I have some test case classes organized in directories
foo_tests
foo_tests1.py
foo_tests2.py
...
bar_tests
bar_tests1.py
...
The test cases look like:
foo_tests1.py:
import unittest
class FooTestsOne(unittest.TestCase):
def test_1():
assert(1=1)
def test_2():
#...
How do you organize test suites out of test case classes like this? There are facilities in unittest for TestLoaders and TestSuite objects but where are they declared and used? What I want is to define certain suites in a separate file that i can run the tests with:
suite1.py
import unittest
import foo_test1
suite = unittest.TestSuite((unittest.makeSuite(foo_tests1.FooTestsOne),
unittest.makeSuite(foo_tests2.FooTeststwo),
))
if __name__ == "__main__":
result = unittest.TextTestRunner(verbosity=2).run(suite())
sys.exit(not result.wasSuccessful())
But this is not the right way to aggregate tests into suites. When I import the testcase class ("import foo_test1") to reference it so I can put it in a larger suite the tests inside are immediately run (during the import). What is the right way to aggregates tests into suites? I need fine grain control as to what tests go into which suites... I've read the TestSuite documentation, but it doesn't seem to provide many examples...
Tests are not supposed to run during import. Maybe you have unittest.main() at the bottom of foo_test1.py?
Your script should work, except that
result = unittest.TextTestRunner(verbosity=2).run(suite())
should be
result = unittest.TextTestRunner(verbosity=2).run(suite)
I don't do tests, but I'd like to start. I have some questions :
Is it ok to use the unittest module for that? From what I understand, the unittest module will run any method starting with test.
if I have a separate directory for the tests ( consider a directory named tests ), how would I import the code I'm testing? Do I need to use the imp module? Here's the directory structure :
src/
tests/
Another good way to start tests with your Python code is use the doctest module, whereby you include tests inside method and class comments. The neat bit is that these serve as code examples, and therefore, partial documentation. Extremely easy to do, too.
It's fine to use unittest. This module will run methods beginning with test for classes which inherit from unittest.TestCase.
There's no need to use the imp module - your test module would just import the code under test as normal. You might need to add the src directory to your path:
import sys
sys.path.append('../src') # OnLinux - use r'..\src' for Windows
This code would be in your test module(s) before any import of your modules.
A better approach is to use the OS environment variable PYTHONPATH.
(Windows) SET PYTHONPATH=path\to\module; python test.py
(Linux) PYTHONPATH=path/to/module; python test.py
An alternative to unittest is nose.
As mentioned by Sebastian P., Mark Pilgrim's Dive Into Python has great chapters on unit testing and test-driven development using Python's unittest module. I used these chapters to get started with testing, myself.
I wrote a blog post describing my approach to importing modules for testing. Note that it solves the shortcoming of Vinay Sajip's approach, which will not work if you call the testing module from anywhere but the directory in which it resides. A reader posted a nice solution in the comments of my blog post as well.
S. Lott hints at a method using PYTHONPATH in Vinay's post; I hope he will expound on it.
I had a bit of problem with the whole seperate directory for tests issue.
The way I solved it was by having a test runner in the source directory with the following code:
import sys, os, re, unittest
# Run all tests in t/
def regressionTest():
path = os.path.abspath(os.path.dirname(sys.argv[0])) + '/t'
files = os.listdir(path)
test = re.compile("^t_.+\.py$", re.IGNORECASE)
files = filter(test.search, files)
filenameToModuleName = lambda f: 't.'+os.path.splitext(f)[0]
moduleNames = map(filenameToModuleName, files)
modules = map(__import__, moduleNames)
modules = map(lambda name: sys.modules[name], moduleNames)
load = unittest.defaultTestLoader.loadTestsFromModule
return unittest.TestSuite(map(load, modules))
suite = regressionTest()
if __name__ == "__main__":
unittest.TextTestRunner(verbosity=2).run(suite)
Then I had a folder named t containing all my tests, named t_<something>.py.
If you need help on getting started with unit testing in Python, I can recommend the official documentation and Dive Into Python among other things.
Few times while browsing tests dir in various Django apps I stumbled across models.py and settings.py files (in django-tagging for example).
But there's no code to be found that syncs test models or applies custom test settings - but tests make use of them just as if django would auto-magically load them. However if I try to run django-tagging's tests: manage.py test tagging, it doesn't do even a single test.
This is exactly what I need right now to test my app, but don't really know how.
So, how does it work?
If you want to run the tests in django-tagging, you can try:
django-admin.py test --settings=tagging.tests.settings
Basically, it uses doctests which are in the tests.py file inside the tests package/directory. The tests use the settings file in that same directory (and specified in the command line to django-admin). For more information see the django documentation on writing doctests.
You mean, "How do I write unit tests in Django?" Check the documentation on testing.
When I've done it, I wrote unit tests in a test/ subdirectory. Make sure the directory has an empty __init__.py file. You may also need a models.py file. Add unit tests that derive from unittest.TestCase (in module unittest). Add the module 'xxxx.test' to your INSTALLED_APPS in settings.py (where 'xxxx' is the base name of your application).
Here's some sample code of mine to get you started:
#!/usr/bin/env python
# http://docs.djangoproject.com/en/dev/topics/testing/
from sys import stderr
import unittest
from django.test.client import Client
from expenses.etl.loader import load_all, load_init
class TestCase(unittest.TestCase):
def setUp(self):
print "setUp"
def testLoading(self):
print "Calling load_init()"
load_init()
print "Calling load_all()"
load_all()
print "Done"
if __name__ == '__main__':
unittest.main()
If you mean, "How do I get data loaded into my unit tests?", then use fixtures, described on the same documentation page.