I am working on a project that has many "unit tests" that have hard dependencies that need to interact with the database and other APIs. The tests are a valuable and useful resource to our team, but they just cannot be ran independently, without relying on the functionality of other services within the test environment. Personally I would call these "functional tests", but this is just the semantics already established within our team.
The problem is, now that we are beginning to introduce more pure unit tests into our code, we have a medley of tests that do or do not have external dependencies. These tests can be ran immediately after checking out code with no requirement to install or configure other tools. They can also be ran in a continuous integration environment like jenkins.
So my question is, how I can denote which is which for a cleaner separation? Is there an existing decorator within unit testing library?
You can define which test should be skipped with the skipIf decorator. In combinations with setting an environmental variable you can skip tests in some environments. An example:
from unittest import skipIf
class MyTest(Testcase):
#skipIf(os.environ.get('RUNON') == 'jenkins', 'Does not run in Jenkins')
def test_my_code(self):
...
Here's another option. You could separate different test categories by directory. If you wanted to try this strategy, it may look something like:
python
-modules
unit
-pure unit test modules
functional
-other unit test modules
In your testing pipeline, you can call your testing framework to only execute the desired tests. For example, with Python's unittest, you could run your 'pure unit tests' from within the python directory with
python -m unittest discover --start-directory ../unit
and the functional/other unit tests with
python -m unittest discover --start-directory ../functional
An advantage of this setup is that your tests are easily categorized and you can do any scaffolding or mocked up services that you need in each testing environment. Someone with a little more Python experience might be able to help you run the tests regardless of the current directory, too.
Related
Is there an example of a self-contained repository showing how to perform SQL unit testing of PyFlink (specifically 1.13.x if possible)?
There is a related SO question here, where it is suggested to use some of the tests from PyFlink itself. The issue I'm running into is that the PyFlink repo assumes that a bunch of things are on the Java classpath and that some Python utility classes are available (they're not distributed via PyPi apache-flink).
I have done the following:
Copied test_case_utils.py and source_sink_utils.py from PyFlink into my project.
Copy an example unit test (this one as suggested by the related SO question.
When I try to run the test, I get an error because the test case cannot determine what version of Avro jars to download (download_apache_avro() fails, because this code tries to evaluate the value of avro.version by running mvn help:evaluate -Dexpression=avro.version)
I then added a dummy pom.xml defining a Maven property of avro.version (with a value of 1.10.0) and my unit test case is loaded.
I now get a new error and my test is skipped:
'flink-table-planner*-tests.jar' is not available. Will skip the related tests.
I don't know how to fix this. I've tried adding flink-table-planner and flink-table-planner-blink dependencies with <type>test-jar</type> to my dummy pom.xml, but it still fails.
This is starting to feel like a real pain to do something that should be trivial: basic TDD of a PyFlink project. Is there a real-world example of a Python project that shows how to set up a testing environment for unit testing SQL with PyFlink?
You can refer to https://github.com/dianfu/pyflink-faq/tree/main/testing which gives an example on how to write unit tests in an external project.
I started learning python as I developed a project about a year ago. Since then the project became somewhat of a (quite large) stable and useful tool for me. The project's arrangement is like so:
main.py
../functions/func1.py
../functions/func2.py
../functions/func3.py
../functions/func4.py
...
../functions/funcN.py
where the main.py file calls the rest of the functions sequentially.
The issue is that I did not write a single unit test for any of the functions. Not one.
I did not pay much attention to testing since at first I was just learning and eventually it got out of hand.
I want to correct this and add the proper unit tests, the question is: which testing method should I use for my project?
I've seen many different methods described:
unittest
Doctest
pytest
nose
tox
unittest2
mock
but I've no idea if one of those is more suited to something like my project than the rest.
unittest which is now just unittest2 is already in python and the most standard, just start with that.
Think of nose as a set of extensions, use it when you want something not already in unittests, it's quite popular as well.
doctests puts unit tests into doc comments, I don't like it too much but use it if you want to.
mock is just a testing paradigm you should use when interacting with interfaces/objects is not trivial
tox runs tests under different python environments.
As an addition, integration tools like travis/jenkins allows you to run tox or sets of unit tests automatically, they're often used for multi-user projects, so everybody can see the test results on each commit.
Is there anything special with using Nose for tests? From what I have heard the reason most people use Nose is..
because it gives you a report
because it shows you the time it took for the tests
How is that any better than using simple Bash like below?
tests.py:
assert test1()
assert test2()
assert test3()
print("No errors")
runtests:
#!/bin/sh
(time python tests.py) > log
return $?
The benefit of using a standard tool is that you are more likely to find third-party tools which build on top of the tool. So for just running a test, it doesn't matter what you use, but as soon as you start having many components in a Jenkins rig, having multiple different tools with different output formats and conventions makes it a real problem to maintain and develop monitoring and reporting.
For shell scripts (which I imagine is part of the question because you used the bash tag and wrote your script in sh), it's not like Nose is "the standard", and if you have multiple tools in different languages, it might not be possible to standardize on a single tool / framework / convention (TAP for Perl, Nose for Python, JUnit or whatever for Java ...)
One benefit which you didn't mention is that the framework takes care of a lot of the footwork for you. A single file with tests could be managed (with some pain) by hand, but once we start talking dozens of files with hundreds or thousands of test cases, you want a decent platform for managing those and let you focus on the actual testing instead of reinventing the wheels that the framework puts there for you to use.
What I want
I would like to create a set of benchmarks for my Python project. I would like to see the performance of these benchmarks change as I introduce new code. I would like to do this in the same way that I test Python, by running the utility command like nosetests and getting a nicely formatted readout.
What I like about nosetests
The nosetests tool works by searching through my directory structure for any functions named test_foo.py and runs all functions test_bar() contained within. It runs all of those functions and prints out whether or not they raised an exception.
I'd like something similar that searched for all files bench_foo.py and ran all contained functions bench_bar() and reported their runtimes.
Questions
Does such a tool exist?
If not what are some good starting points? Is some of the nose source appropriate for this?
nosetests can run any type of test, so you can decide if they test functionality, input/output validity etc., or performance or profiling (or anything else you'd like). The Python Profiler is a great tool, and it comes with your Python installation.
import unittest
import cProfile
class ProfileTest(unittest.TestCase):
test_run_profiler:
cProfile.run('foo(bar)')
cProfile.run('baz(bar)')
You just add a line to the test, or add a test to the test case for all the calls you want to profile, and your main source is not polluted with test code.
If you only want to time execution and not all the profiling information, timeit is another useful tool.
The wheezy documentation has a good example on how to do this with nose. The important part if you just want to have the timings is to use options -q for quiet run, -s for not capturing the output (so you will see the output of the report) and -m benchmark to only run the 'timing' tests.
I recommend using py.test for testing over. To run the example from wheezy with that, change the name of the runTest method to test_bench_run and run only this benchmark with:
py.test -qs -k test_bench benchmark_hello.py
(-q and -s having the same effect as with nose and -k to select the pattern of the test names).
If you put your benchmark tests in file in a separate file or directory from normal tests they are of course more easy to select and don't need special names.
I have several thousand tests that I want to run in parallel. The tests are all compiled binaries that give a return code of 0 or non-zero (on failure). Some unknown subsets of them try to use the same resources (files, ports, etc). Each test assumes that it is running independently and just reports a failure if a resources isn't available.
I'm using Python to launch each test using the subprocess module, and that works great serially. I looked into Nose for parallelizing, but I need to autogenerate the tests (to wrap each of the 1000+ binaries into Python class that uses subprocess) and Nose's multiprocessing module doesn't support parallelizing autogenerated tests.
I ultimately settled on PyTest because it can run autogenerated tests on remote hosts over SSH with the xdist plugin.
However, as far as I can tell, it doesn't look like xdist supports any kind of control of how the tests get distributed. I want to give it a pool of N machines, and have one test run per machine.
Is what I want possible with PyTest/xdist? If not, is there a tool out there that can do what I'm looking for?
I am not sure if this would help. But if you know ahead of time how you want to divide up your tests, instead of having pytest distribute your tests, you could use your continuous integration server to call a different run of pytest for each different machine. Using -k or -m to select a subset of tests, or simply specifying different test dir paths, you could control which tests are run together.