We started writing our functional and unit test cases in python using nose framework. We started learning python while writing these tests. Since there are lot of dependencies between our test classes/functions, we decided to use proboscis framework on top of nose to control order of execution.
We have quite a few 'print' statements in our tests and proboscis seems to be ignoring these! Tests are running in expected order and testing them all but not printing our print statement data to console. Any idea what we are missing here?
BTW, we stopped deriving our classes from 'unittest.TestCase' once we moved to proboscis and decorated all classes and their member functions with #test.
Note: According to the Proboscis documentation "unused arguments get passed along to Nose or the unittest module", so the following should apply to Proboscis by replacing nosetests with python run_tests.py.
As #Wooble has mentioned in his comment, by default nose captures stdout and only displays it for failed tests. You can override this behaviour with the nosetests -s or --nocapture switch:
$ nosetests --nocapture
Like #Wooble also mentions in his comment, I recommend using the logging module instead of print. Then you only need to pass nosetests the -l DEBUG or --debug=DEBUG switch, where DEBUG is replaced by a comma separated list of the names of the loggers you want to display, to enable displaying of the logging output from your modules:
$ nosetests --debug=your-logger-name
Related
By default, pytest inflates the error traceback massively and printly some information into sysout stream that are redundant: Considering that I'm using PyCharm, it is really obfuscating to see code snippet out of context, while they are already available in the IDE & debugging interface.
As a result, I intend to set pytest traceback to native permanently. However, according to the documentation, the only way to do so is to add extra command line argument when launching the test runner:
-tb=native
I would like to make my test to always use the native traceback regardless of how it was run. Is it possible to use a TestCase API to do so?
Thanks a lot for your help.
You can add this option to the pytest.ini file and it would be automatically picked by pytest. For your specific case, a pytest.ini with following contents should work:
[pytest]
addopts = --tb=native
Note the double hyphens with tb; I am using pytest 4.6.4 and that is how it works for me.
Also, Refer pytest docs for another alternative by modifying PYTEST_ADDOPTS env variable.
I'm not sure how you can do this using pytest, nor am I familiar with this package. With that being said, you can always create a bash function to accomplish this:
function pytest() {
pytest -tb=native "$#"
}
The "$#" symbol will pass all arguments following pyt to the function (kind of like *args in python), so running pyt arg1 arg2 ... argn will be the same as running
pytest -tb=native arg1 arg2 ... argn
If you are unfamiliar with creating bash shortcuts, see this question.
Update
I misunderstood and thought OP was calling pytest from the cli. Instead of creating the pyt function, if you override pytest directly, PyCharm might invoke your bash version of it instead (I'm not really sure though).
That being said, yaniv's answer seems superior to this, if it works.
Everything works fine with PyCharm and pytest, except if I have failing tests, then it duplicates the error output:
One of actual failures if the red one and other is white. This is really annoys, and I haven't found any way to disable such behaviour.
There is an option to disable log via py.test, however it will disable all logging.
Note: everything works as expected if I run python -m pytest test.py.
I think that is a feature not a bug. The top level is being emitted during the testing which allows you to review the failure before the testing is complete. The second copy of the results is the summary which effectively removes any of the text that was showing test progress.
You can easily view just part of the test output by clicking on the test hierarchy:
The duplicated output can be elimited by running pytest with the -q or --quiet parameter.
You can configure the parameter to be applied to all PyCharm pytest tests by setting it in Edit Configurations --> Templates --> Python Tests --> pytest --> Additional Arguments.
This will then apply those arguments to all new run configurations. If you have a bunch of existing test run configurations, deleting all of them and then re-generating them by running a test or tests using the gutter icon is the quickest way to reset the output.
Typically I run my python unit test with:
python -m unittest test.<module-name>
I would like to debug my tests using ipdb but I cannot figure out how to invoke theunittest module in manner similar to the command above.
The directory structure is:
base/src for the source code
base/test for the test code
Tests run from the base directory.
The preference is to 'run' & 'debug' test cases in the same manner, specifically the preference is to debug with ipdb in a similar manner to the python command above.
FYI
Interpreter is python2
I don't know how to do this with the unittest module from the standardlib, but the testrunner pytest has at least pdb (not ipdb) support. Just invoke it with
pytest --pdb
and it drops into pdb with the first failure.
We have recently switched to py.test for python testing (which is fantastic btw). However, I'm trying to figure out how to control the log output (i.e. the built-in python logging module). We have pytest-capturelog installed and this works as expected and when we want to see logs we can pass --nologcapture option.
However, how do you control the logging level (e.g. info, debug etc.) and also filter the logging (if you're only interested in a specific module). Is there existing plugins for py.test to achieve this or do we need to roll our own?
Thanks,
Jonny
Installing and using the pytest-capturelog plugin could satisfy most of your pytest/logging needs. If something is missing you should be able to implement it relatively easily.
As Holger said you can use pytest-capturelog:
def test_foo(caplog):
caplog.setLevel(logging.INFO)
pass
If you don't want to use pytest-capturelog you can use a stdout StreamHandler in your logging config so pytest will capture the log output. Here is an example basicConfig
logging.basicConfig(level=logging.DEBUG, stream=sys.stdout)
A bit of a late contribution, but I can recommend pytest-logging for a simple drop-in logging capture solution. After pip install pytest-logging you can control the verbosity of the your logs (displayed on screen) with
$ py.test -s -v tests/your_test.py
$ py.test -s -vv tests/your_test.py
$ py.test -s -vvvv tests/your_test.py
etc... NB - the -s flag is important, without it py.test will filter out all the sys.stderr information.
Pytest now has native support for logging control via the caplog fixture; no need for plugins.
You can specify the logging level for a particular logger or by default for the root logger:
import pytest
def test_bar(caplog):
caplog.set_level(logging.CRITICAL, logger='root.baz')
Pytest also captures log output in caplog.records so you can assert logged levels and messages. For further information see the official documentation here and here.
A bit of an even later contribution: you can try pytest-logger. Novelty of this plugin is logging to filesystem: pytest provides nodeid for each test item, which can be used to organize test session logs directory (with help of pytest tmpdir facility and it's testcase begin/end hooks).
You can configure multiple handlers (with levels) for terminal and filesystem separately and provide own cmdline options for filtering loggers/levels to make it work for your specific test environment - e.g. by default you can log all to filesystem and small fraction to terminal, which can be changed on per-session basis with --log option if needed. Plugin does nothing by default, if user defines no hooks.
I am using py.test for unit testing my python program. I wish to debug my test code with the python debugger the normal way (by which I mean pdb.set_trace() in the code) but I can't make it work.
Putting pdb.set_trace() in the code doesn't work (raises IOError: reading from stdin while output is captured). I have also tried running py.test with the option --pdb but that doesn't seem to do the trick if I want to explore what happens before my assertion. It breaks when an assertion fails, and moving on from that line means terminating the program.
Does anyone know a way to get debugging, or is debugging and py.test just not meant to be together?
it's real simple: put an assert 0 where you want to start debugging in your code and run your tests with:
py.test --pdb
done :)
Alternatively, if you are using pytest-2.0.1 or above, there also is the pytest.set_trace() helper which you can put anywhere in your test code. Here are the docs. It will take care to internally disable capturing before sending you to the pdb debugger command-line.
I found that I can run py.test with capture disabled, then use pdb.set_trace() as usual.
> py.test --capture=no
============================= test session starts ==============================
platform linux2 -- Python 2.5.2 -- pytest-1.3.3
test path 1: project/lib/test/test_facet.py
project/lib/test/test_facet.py ...> /home/jaraco/projects/project/lib/functions.py(158)do_something()
-> code_about_to_run('')
(Pdb)
The easiest way is using the py.test mechanism to create breakpoint
http://pytest.org/latest/usage.html#setting-a-breakpoint-aka-set-trace
import pytest
def test_function():
...
pytest.set_trace() # invoke PDB debugger and tracing
Or if you want pytest's debugger as a one-liner, change your import pdb; pdb.set_trace() into import pytest; pytest.set_trace()
Similar to Peter Lyon's answer, but with the exact code you need for pytest, you can add the following to the bottom of your pytest module (my_test_module.py) :
if __name__ == "__main__":
pytest.main(["my_test_module.py", "-s"])
Then you can invoke the debugger from the command line:
pdb3 my_test_module.py
Boom. You're in the debugger and able to enter debugger commands. This method leaves your test code un-littered with set_trace() calls and will run inside pytest 'normally'.
Simply use: pytest --trace test_your_test.py.
This will invoke the Python debugger at the start of the test
I'm not familiar with py.test, but for unittest, you do the following. Maybe py.test is similar:
In your test module (mytestmodule.py):
if __name__ == "__main__":
unittest.main(module="mytestmodule")
Then run the test with
python -m pdb mytestmodule.py
You will get an interactive pdb shell.
Looking at the docs, it looks like py.test has a --pdb command line option:
https://docs.pytest.org/en/7.2.x/reference/reference.html#command-line-flags
Add and remove breakpoints without editing source files
Although you can add breakpoints by adding breakpoint() or set_trace() statements to your code, there are two issues with this approach:
Firstly, once you have started running your code, there is no way to remove your breakpoint. I often find that once I start running my code and reach an initial breakpoint, I want to place another one and remove the initial breakpoint. After breakpoint() drops me into the debugger I can add additional breakpoints, but I can't remove the initial one. Although this can be mitigated somewhat by putting the initial breakpoint statement higher up, if you have parametrised tests then even that is limited. I may find myself repeating cont very often.
Secondly, it requires changes to the source code. You need to remember to remove all breakpoint() commands before committing any code to version control, you have to remove them before switching branches, etc. I sometimes find I want to use the debugger to compare test runs between two branches, and having to edit the source code to add a breakpoint every time makes that a considerably slower and more error-prone exercise. I may even want to add a breakpoint in a library I'm calling, in which case the file I'm editing may not even me in my git repository but somewhere deep in my conda environment, increasing the risk of forgetting to remove it. Editing files to add break points is, in my humble opinion, ugly.
To add and remove breakpoints interactively without editing any source files, you can evoke pytest as follows (in the bash shell):
python -mipdb $(type -p pytest) -s test_fileset.py
The -s flag is crucial here, because it stops pytest from messing with stdin and stdout, and when running inside the debugger, pytest will fail to mess with stdin and stdout and everything will go wrong. The exact calling syntax will be different for different shells.