What is actually the difference between using unit tests and normal tests?
By normal tests I mean, using the if statement for example to determine if the calculation is equal to the desired answer if it returns false, we can raise AssertionError
Let's use a simple piece of code as example:
def square(a):
return a*a;
Tests that aim at finding bugs in this function in isolation are unit-tests. This is independent of how you actually implement them: If you just write an if statement like if (square(3) != 9) and raise an AssertionError as you say, then it is a unit-test. If instead you implement the same test using unittest by calling assertEqual, then it is also a unit-test.
In other words, whether or not you use a (so-called) unit-test framework is not a criterion for whether your tests are unit-test or not. In fact, despite the names of the frameworks ('unittest' in Python, 'JUnit' in Java, ...) these frameworks can be used for unit-tests as well as for other tests, like, integration-tests. The names of those frameworks are therefore a bit misleading.
So much for the original question about the difference between unit-tests and normal tests. In one of the comments you make it clearer that you actually want to know what is better: To use or not to use a test framework. And, the answer is clear: Definitely go for a test framework.
After writing only a few tests 'by hand', that is, without a test framework, you will see that there is a lot of duplication in your test code: You compare results with if - that is not so much different. But then, in the success case you write some 'passed' message, with the test name, in the failure case you write a 'failed' messages, again with test name and in this case also with some extra information about what was the actual and what was the expected result. And - did you think about the case that the code under test exits with an exception? So, you also have to wrap it with a try/catch block to make sure unexpected exceptions lead to a 'failed' result, again with some helpful diagnostic information.
And so on... All this and more is taken care of by the test frameworks for you.
From your question, you assume there are two kinds of test types: unit tests and normal test. In fact, there is a lot of types of tests. The unit test is one of them. You can read here other kinds of tests.
If we assume 'normal test' is 'black box test'. It is testing software without source code. For unit tests, there is another part of the software that tests the integrity of the source code itself. What does that mean? You wrote a function that sums two numbers. For the unit test, you are writing another function to check whether your first function works
# Your Function
def my_sum_function(x,y):
return(x+y)
# Your Unit Test Function
def test_sum():
assert my_sum_function([1, 2]) == 3, "Should be 6"
Not only for Python but also for all programming language logic behind the test is the same.
For quality control. If you are coding an app called "app1" with a function f1 and an other function f2 (f2 is using f1) inside you own library
Without unit test:
You can make a change in function f1 test it, it will work in f2 but you will not notice that in you program "app0" you wrote 2 weeks ago it will make you app crash because in your old app "app0" you have a function f3 and f3 is also using f1....
With unit test:
You can catch that type of nasty bug because the test for the function f3 will not pass anymore and you IDE will tell you that when you change the function f1 in "app1"
I hope it make sense (english is not my native language)
Related
Question is, is there a way mark unit tests by different categories: unit, integration, "dangerous_integration". Then from say "Team City" setup, auto skip "dangerous_integration"?
For tests which is marked "Dangerous" - they should not be run automatically. Those need run manually, preferably even on debugger with very close watch?
For python, here's an example where a class I have that can send live trades to crypto exchanges. I essentially shunt all test methods with a "return" as first line of test method to avoid situation where we send live trades out to the street unintentionally.
import unittest
class BinanaceConnectorTests(unittest.TestCase):
def setUp(self):
...
def tearDown(self):
...
def testSendRealOrders(self):
return <-- I hardcode return here so nobody accidentally
... actual implementation shunted by above return ...
Is it the proper way of doing this? I am using Visual Studio. I remember in C# with xunit, there's way you can mark unit tests by different categories.
I am thinking there isnt such thing as "dangerous_integration" tests - if you need such thing, separate those tests out to a manual python script and simply don't mark them as unit tests of any kind. That'd be safest.
The scenarios for selective test execution that you have presented (like dangerous_integration) should probably better be addressed in a different way, as was already discussed in the comments. There are, however, situations where you want certain tests to be excluded in the normal case. For example, you might have some long-running tests which you only want to execute occasionally. Or, some tests produce outputs beyond simple 'passed' or 'failed' results that require thorough analysis, which you also only want to run at times.
One approach can be to group these tests into classes of their own and use this to include or exclude them from the respective runs. Or, if it seems more appropriate to have these tests implemented together with other tests, you could use some naming convention that indicates to which category a test belongs. You have the option to selectively run tests that are grouped or named in certain ways as is described in the Python unittest manual: https://docs.python.org/3.7/library/unittest.html. For example, the attribute testNamePatterns of unittest.TestLoader could be used for the selection of tests based on their names.
I have the following scenario:
I have a list of "dangerous patterns"; each is a string that contains dangerous characters, such as "%s with embedded ' single quote", "%s with embedded \t horizontal tab" and similar (it's about a dozen patterns).
Each test is to be run
once with a vanilla pattern that does not inject dangerous characters (i.e. "%s"), and
once for each dangerous pattern.
If the vanilla test fails, we skip the dangerous pattern tests on the assumption that they don't make much sense if the vanilla case fails.
Various other constraints:
Stick with the batteries included in Python as far as possible (i.e. unittest is hopefully enough, even if nose would work).
I'd like to keep the contract of unittest.TestCase as far as possible. I.e. the solution should not affect test discovery (which might find everything that starts with test, but then there's also runTest which may be overridden in the construction, and more variation).
I tried a few solutions, and I am not happy with any of them:
Writing an abstract class causes unittest to try and run it as a test case (because it quacks like a test case). This can be worked around, but the code is getting ugly fast. Also, a whole lot of functions needs to be overridden, and for several of them the documentation is a bit unclear about what properties need to be implemented in the base class. Plus test discovery would have to be replicated, which means duplicating code from inside unittest.
Writing a function that executes the tests as SubTests, to be called from each test function. Requires boilerplate in every test function, and gives just a single test result for the entire series of tests.
Write a decorator. Avoids the test case discovery problem of the abstract class approach, but has all the other problems.
Write a decorator that takes a TestCase and returns a TestSuite. This worked best so far, but I don't know whether I can add the same TestCase object multiple times. Or whether TestCase objects can be copied or not (I have control over all of them but I don't know what the base class does or expects in this regard).
What's the best approach?
So I'm trying to decide the way to plan and organize a testing suite for my python project but I have a doubt of when a unit test is no longer a unit test and I would love to have some feedback from the comunity.
If I understand correctly:
A Unit test test a minimal part of your code, be if a function/method that does one and only one simple thing, even if it has several use cases.
An Integration test tests that two or more units of you code that are executed under the same context, environment, etc (but trying to keep it to a minimmum of units per integration test) work well together and not only by themselves.
My doubt is: say I have a simple function that performs a HTTP request and returns the content of such request, be it HTML, JSON, etc, it doesn't matter, the fact is that the function is very, very simple but requests information from an external source, like:
import requests
def my_function(arg):
# do something very simple with `arg`, like removing spaces or the simplest thing you can imagine
return requests.get('http://www.google.com/' + arg).content
Now this is a very stupid example, but my doubt is:
Given that this function is requesting information from an external source, when you write a test for it, can you still consider such test a Unit test?
UPDATE: The test for my_function() would stub out calls to the external source so that it doesn't depend on network/db/filesystem/etc so that it's isolated. But the fact that the function that is being tested depends on external sources when running, for example, in production.
Thanks in advance!! :)
P.S.: Of course Maybe I'm not understading 100% de purposes of Unit and Integration testing, so, if I'm mistaken, please point me out where, I'll appreciate it.
Based on your update:
The test for my_function() would stub out calls to the external source
so that it doesn't depend on network/db/filesystem/etc so that it's
isolated. But the fact that the function that is being tested depends
on external sources when running, for example, in production.
As long as the external dependencies are stubbed out during your test then yes you can call it a Unit Test. It's a Unit Test based on how the unit under test behaves in your test suite rather than how the unit behaves in production.
Based on your original question:
Given that this function is requesting information from an external
source, when you write a test for it, can you still consider such test
a Unit test?
No, any test for code that touches or depends on things external to the unit under test is an integration test. This includes the existence of any web, file system, and database requests.
If your code is not 100% isolated from it's dependencies and not 100% reproducible without other components then it is an integration test.
For your example code to be properly unit tested you would need to mock out the call to google.com.
With the code calling google.com, your test would fail if google went down or you lost connection to the internet (ie the test is not 100% isolated). Your test would also fail if the behavior of google changed (ie the test is not 100% reproducable).
I don't think this is an integration test since it doesn't make use of different parts of your application. The function does really one thing and tests for it can be called unit.
On the other side, this particular function has an external dependency and you don't want to be dependent on the network in your tests. This is where mocking would really help a lot.
In other words, isolate the function and make unit tests for it.
Also, make integration tests that would use a more high-level approach and would test parts of your application which call my_function().
Whether or not your code is tested in isolation is not a distinguishing criterion whether your test is a unit test or not. For example, you would (except in very rare cases) not stub standard library functions like sin(x). But, if you don't stub sin(x) that does not make your test an integration test.
When is a test an integration test? A test is an integration test if your goal with the test is to find bugs on integration level. That means, with an integration test you want to find out whether the interaction between two (or more) components is based on the same assumptions on both (all) sides.
Mocking, however, is an orthogonal technique that can be combined with almost all kinds of tests. (In an integration test, however, you can not mock the partners of the interaction of which you want to test - but you can mock additional components).
Since mocking typically causes some effort, it must bring a benefit, like:
significantly speeding up your tests
testing although some software part is not ready yet or buggy
testing exceptional cases that are hard to set up in an integrated software
getting rid of nondeterministic behaviour like timing or randomness
...
If, however, mocking does not solve a real problem, you might be better off using the depended-on component directly.
I've never written a proper test until now, only small programs that I would dispose of after the test succeeded. I was looking through Python's unittest module and tutorials around the web, but something's not clear to me.
How much should one TestCase cover? I've seen examples on the web that have TestCase classes with only one method, as well as classes that test almost the entire available functionality.
In my case, I'm trying to write a test for a simple bloom filter. How do you think I should organize my test cases?
To put it simple: one unit test should cover single feature of your program. That's all there is to say. That's why they're called unit tests.
Of course, what we understand by feature may vary. Think about smallest parts of your program that might break or not work as expected. Think about business requirements of your code. Those are parts that you want each to be covered by dedicated unit test.
Usually, unit tests are small, isolated and atomic. They should be easy to understand, they should fail/pass independently from one another, and should execute fast. Fairly good indication of proper unit tests is single assertion - if you find yourself writing more, you probably test too much and it's a sign you need more than one test for given feature. However, this is not a strict rule - the more complex code is involved, the more complex unit tests tend to be.
When writing tests, it's easy to split your code functionality and test those separated parts (this should give you the idea of atomicity of your tests). For example, if you have a method that verifies input then calls a service and finally returns result, you usually want to have all three (verify, call, return) steps covered.
I would create one TestCase with several test methods. A bloom filter has simple semantics, so only one TestCase. I usually add a TestCase per feature.
Example
Let's say you have a hypothetical API like this:
import foo
account_name = foo.register()
session = foo.login(account_name)
session.do_something()
The key point being that in order to do_something(), you need to be registered and logged in.
Here's an over-simplified, first-pass, suite of unit tests one might write:
# test_foo.py
import foo
def test_registration_should_succeed():
foo.register()
def test_login_should_succeed():
account_name = foo.register()
foo.login(account_name)
def test_do_something_should_succeed():
account_name = foo.register()
session = foo.login(account_name)
session.do_something()
The Problem
When registration fails, all the tests fail and that makes it non-obvious where
the real problem is. It looks like everything's broken, but really only one, crucial, thing is broken. It's hard to find that once crucial thing unless you are familiar with all the tests.
The Question
How do you structure your unit tests so that subsequent tests aren't executed when core functionality on which they depend fails?
Ideas
Here are possible solutions I've thought of.
Manually detect failures in each test and and raise SkipTest. - Works, but a lot of manual, error-prone work.
Leverage generators to not yield subsequent tests when the primary ones fail. - Not sure if this would actually work (because how do I "know" the previously yielded test failed).
Group tests into test classes. E.g., these are all the unit tests that require you to be logged in. - Not sure this actually addresses my issue. Wouldn't there be just as many failures?
Rather than answering the explicit question, I think a better answer is to use mock objects. In general, unit tests should not require accessing external databases (as is presumably required to log in). However, if you want to have some integration tests that do this (which is a good idea), then those tests should test the integration aspects, and your unit tests should tests the unit aspects. I would go so far as to keep the integration tests and the unit tests in separate files, so that you can quickly run the unit tests on a very regular basis, and run the integration tests on a slightly less regular basis (although still at least once a day).
This problem indicates that Foo is doing to much. You need to separate concerns. Then testing will become easy. If you had a Registrator, a LoginHandler and a DoSomething class, plus a central controller class that orchestrated the workflow, then everything could be tested separately.