I have the following scenario:
I have a list of "dangerous patterns"; each is a string that contains dangerous characters, such as "%s with embedded ' single quote", "%s with embedded \t horizontal tab" and similar (it's about a dozen patterns).
Each test is to be run
once with a vanilla pattern that does not inject dangerous characters (i.e. "%s"), and
once for each dangerous pattern.
If the vanilla test fails, we skip the dangerous pattern tests on the assumption that they don't make much sense if the vanilla case fails.
Various other constraints:
Stick with the batteries included in Python as far as possible (i.e. unittest is hopefully enough, even if nose would work).
I'd like to keep the contract of unittest.TestCase as far as possible. I.e. the solution should not affect test discovery (which might find everything that starts with test, but then there's also runTest which may be overridden in the construction, and more variation).
I tried a few solutions, and I am not happy with any of them:
Writing an abstract class causes unittest to try and run it as a test case (because it quacks like a test case). This can be worked around, but the code is getting ugly fast. Also, a whole lot of functions needs to be overridden, and for several of them the documentation is a bit unclear about what properties need to be implemented in the base class. Plus test discovery would have to be replicated, which means duplicating code from inside unittest.
Writing a function that executes the tests as SubTests, to be called from each test function. Requires boilerplate in every test function, and gives just a single test result for the entire series of tests.
Write a decorator. Avoids the test case discovery problem of the abstract class approach, but has all the other problems.
Write a decorator that takes a TestCase and returns a TestSuite. This worked best so far, but I don't know whether I can add the same TestCase object multiple times. Or whether TestCase objects can be copied or not (I have control over all of them but I don't know what the base class does or expects in this regard).
What's the best approach?
Related
What is actually the difference between using unit tests and normal tests?
By normal tests I mean, using the if statement for example to determine if the calculation is equal to the desired answer if it returns false, we can raise AssertionError
Let's use a simple piece of code as example:
def square(a):
return a*a;
Tests that aim at finding bugs in this function in isolation are unit-tests. This is independent of how you actually implement them: If you just write an if statement like if (square(3) != 9) and raise an AssertionError as you say, then it is a unit-test. If instead you implement the same test using unittest by calling assertEqual, then it is also a unit-test.
In other words, whether or not you use a (so-called) unit-test framework is not a criterion for whether your tests are unit-test or not. In fact, despite the names of the frameworks ('unittest' in Python, 'JUnit' in Java, ...) these frameworks can be used for unit-tests as well as for other tests, like, integration-tests. The names of those frameworks are therefore a bit misleading.
So much for the original question about the difference between unit-tests and normal tests. In one of the comments you make it clearer that you actually want to know what is better: To use or not to use a test framework. And, the answer is clear: Definitely go for a test framework.
After writing only a few tests 'by hand', that is, without a test framework, you will see that there is a lot of duplication in your test code: You compare results with if - that is not so much different. But then, in the success case you write some 'passed' message, with the test name, in the failure case you write a 'failed' messages, again with test name and in this case also with some extra information about what was the actual and what was the expected result. And - did you think about the case that the code under test exits with an exception? So, you also have to wrap it with a try/catch block to make sure unexpected exceptions lead to a 'failed' result, again with some helpful diagnostic information.
And so on... All this and more is taken care of by the test frameworks for you.
From your question, you assume there are two kinds of test types: unit tests and normal test. In fact, there is a lot of types of tests. The unit test is one of them. You can read here other kinds of tests.
If we assume 'normal test' is 'black box test'. It is testing software without source code. For unit tests, there is another part of the software that tests the integrity of the source code itself. What does that mean? You wrote a function that sums two numbers. For the unit test, you are writing another function to check whether your first function works
# Your Function
def my_sum_function(x,y):
return(x+y)
# Your Unit Test Function
def test_sum():
assert my_sum_function([1, 2]) == 3, "Should be 6"
Not only for Python but also for all programming language logic behind the test is the same.
For quality control. If you are coding an app called "app1" with a function f1 and an other function f2 (f2 is using f1) inside you own library
Without unit test:
You can make a change in function f1 test it, it will work in f2 but you will not notice that in you program "app0" you wrote 2 weeks ago it will make you app crash because in your old app "app0" you have a function f3 and f3 is also using f1....
With unit test:
You can catch that type of nasty bug because the test for the function f3 will not pass anymore and you IDE will tell you that when you change the function f1 in "app1"
I hope it make sense (english is not my native language)
What's the best practice to provide the description of a test script in python?
Obviously I can put the comments below the test case, but wanted to know if there's any standard practice (any methods I should write) to provide the description of the test case (detailed information on what the test case is supposed to do)?
Is this how you would put the test description?:
Class TestFoo:
def testfoo1():
"""
test description:
step1:
step2:
"""
Any suggestions/references would be appreciated.
The test method docstring is the standard place to put that information If you are using python's unittest module. unittest will use that docstring to format output, etc.
In the unittest framework, you have the
shortDecription method:
shortDescription()
Returns a description of the test, or None if no description has been provided. The default implementation of this method returns the first line of the test method’s docstring, if available.
So, in fact, using the method docstring is a fine place. You may have to inherit from TestCase in your class declaration for the runner to work like that, though.
For best practice: name the test case (class) and the test methods in a concise but useful fashion which is sufficient for developers to have a high level idea of where something is going wrong, should that particular test fail. A prerequisite to this is each test method should only be testing one thing, rather than asserting on a whole bunch of different things.
With sensible test names, usually a docstring with "detailed information on what the test case is supposed to do" would not be necessary. If you have existing large tests which check many things, you may want to split them up into a bunch of smaller tests which each assert on one and only one individual thing.
The best way would be for the name of the class to be descriptive enough for a description not to be needed.
Barring that, the docstring is the best approach.
I've never written a proper test until now, only small programs that I would dispose of after the test succeeded. I was looking through Python's unittest module and tutorials around the web, but something's not clear to me.
How much should one TestCase cover? I've seen examples on the web that have TestCase classes with only one method, as well as classes that test almost the entire available functionality.
In my case, I'm trying to write a test for a simple bloom filter. How do you think I should organize my test cases?
To put it simple: one unit test should cover single feature of your program. That's all there is to say. That's why they're called unit tests.
Of course, what we understand by feature may vary. Think about smallest parts of your program that might break or not work as expected. Think about business requirements of your code. Those are parts that you want each to be covered by dedicated unit test.
Usually, unit tests are small, isolated and atomic. They should be easy to understand, they should fail/pass independently from one another, and should execute fast. Fairly good indication of proper unit tests is single assertion - if you find yourself writing more, you probably test too much and it's a sign you need more than one test for given feature. However, this is not a strict rule - the more complex code is involved, the more complex unit tests tend to be.
When writing tests, it's easy to split your code functionality and test those separated parts (this should give you the idea of atomicity of your tests). For example, if you have a method that verifies input then calls a service and finally returns result, you usually want to have all three (verify, call, return) steps covered.
I would create one TestCase with several test methods. A bloom filter has simple semantics, so only one TestCase. I usually add a TestCase per feature.
Example
Let's say you have a hypothetical API like this:
import foo
account_name = foo.register()
session = foo.login(account_name)
session.do_something()
The key point being that in order to do_something(), you need to be registered and logged in.
Here's an over-simplified, first-pass, suite of unit tests one might write:
# test_foo.py
import foo
def test_registration_should_succeed():
foo.register()
def test_login_should_succeed():
account_name = foo.register()
foo.login(account_name)
def test_do_something_should_succeed():
account_name = foo.register()
session = foo.login(account_name)
session.do_something()
The Problem
When registration fails, all the tests fail and that makes it non-obvious where
the real problem is. It looks like everything's broken, but really only one, crucial, thing is broken. It's hard to find that once crucial thing unless you are familiar with all the tests.
The Question
How do you structure your unit tests so that subsequent tests aren't executed when core functionality on which they depend fails?
Ideas
Here are possible solutions I've thought of.
Manually detect failures in each test and and raise SkipTest. - Works, but a lot of manual, error-prone work.
Leverage generators to not yield subsequent tests when the primary ones fail. - Not sure if this would actually work (because how do I "know" the previously yielded test failed).
Group tests into test classes. E.g., these are all the unit tests that require you to be logged in. - Not sure this actually addresses my issue. Wouldn't there be just as many failures?
Rather than answering the explicit question, I think a better answer is to use mock objects. In general, unit tests should not require accessing external databases (as is presumably required to log in). However, if you want to have some integration tests that do this (which is a good idea), then those tests should test the integration aspects, and your unit tests should tests the unit aspects. I would go so far as to keep the integration tests and the unit tests in separate files, so that you can quickly run the unit tests on a very regular basis, and run the integration tests on a slightly less regular basis (although still at least once a day).
This problem indicates that Foo is doing to much. You need to separate concerns. Then testing will become easy. If you had a Registrator, a LoginHandler and a DoSomething class, plus a central controller class that orchestrated the workflow, then everything could be tested separately.
I'm currently writing a set of unit tests for a Python microblogging library, and following advice received here have begun to use mock objects to return data as if from the service (identi.ca in this case).
However, surely by mocking httplib2 - the module I am using to request data - I am tying the unit tests to a specific implementation of my library, and removing the ability for them to function after refactoring (which is obviously one primary benefit of unit testing in the firt place).
Is there a best of both worlds scenario? The only one I can think of is to set up a microblogging server to use only for testing, but this would clearly be a large amount of work.
You are right that if you refactor your library to use something other than httplib2, then your unit tests will break. That isn't such a horrible dependency, since when that time comes it will be a simple matter to change your tests to mock out the new library.
If you want to avoid that, then write a very minimal wrapper around httplib2, and your tests can mock that. Then if you ever shift away from httplib2, you only have to change your wrapper. But notice the number of lines you have to change is the same either way, all that changes is whether they are in "test code" or "non-test code".
Not sure what your problem is. The mock class is part of the tests, conceptually at least. It is ok for the tests to depend on particular behaviour of the mock objects that they inject into the code being tested. Of course the injection itself should be shared across unit tests, so that it is easy to change the mockup implementation.