The results of python tests are cached for only a short time in the VSCode MS python extension.
Whether they succeeded or failed is only cached for a short while, and then they revert back to question marks again.
How to retain the results of the tests?
I think it has to do with this some logs that I see in the Output for "Python Test Log", which shows many (~30) of these lines:
python /home/.../.vscode/extensions/ms-python.python-2020.6.89148/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /home/projectdir --cache-clear -s
In particular the --cache-clear is suspicious, however I don't know what is triggering these outputs.
So my question is: how to remove --cache-clear from the call? Or maybe something else is going on entirely?
I too found myself in need of disabling this vscode behaviour.
I'm using pytest with the pytest-html plugin, and after all tests have run, as soon as i change a test file and save, this command is automatically run by vscode:
python c:\Users\username\.vscode\extensions\ms-python.python-2021.6.944021595\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir c:\Users\username\app\app-test-automation -s --cache-clear --html=report.html --self-contained-html tests
Which inevitably overwrite the .html test report.
I haven't found a way to remove the --cache-clear from the call.
BUT
You can disable the auto discovery of test (guilty of rerunning the command with the --cache-clear flag) by changing the following vscode setting:
Picture of the setting from the vscode settings menu
I am currently developing some tests using python py.test / unittest that, via subprocess, invoke another python application (so that I can exercise the command line options, and confirm that the tool is installed correctly).
I would like to be able to run the tests in such a way that I can get a view of the code coverage metrics (using coverage.py) for the target application using pytest_cov. By default this does not work as the code coverage instrumentation does not apply to code invoked with subprocess.
Code Coverage of the code does work if I update the tests to directly invoke the entry class for the target application (rather than running via the command line).
Ideally I want to have a single set of code which can be run in two ways:
If code coverage monitoring is not enabled then use command line
Otherwise execute the main class of the target application.
Which leads to my question(s):
Is it possible for a python unit test to determine if it is being run with code coverage enabled?
Otherwise: is there any easy way to pass a command line flag from the pytest invocation that can be used to set the mode within the code.
Coverage.py has a facility to automatically measure coverage in sub-processes that are spawned: http://coverage.readthedocs.io/en/latest/subprocess.html
Coverage sets three environment flags when running tests: COV_CORE_SOURCE, COV_CORE_CONFIG and COV_CORE_DATAFILE.
So you can use a simple if-statement to verify whether the current test is being run with coverage enabled:
import os
if "COV_CORE_SOURCE" in os.environ:
# do what yo need to do when Coverage is enabled
Typically I run my python unit test with:
python -m unittest test.<module-name>
I would like to debug my tests using ipdb but I cannot figure out how to invoke theunittest module in manner similar to the command above.
The directory structure is:
base/src for the source code
base/test for the test code
Tests run from the base directory.
The preference is to 'run' & 'debug' test cases in the same manner, specifically the preference is to debug with ipdb in a similar manner to the python command above.
FYI
Interpreter is python2
I don't know how to do this with the unittest module from the standardlib, but the testrunner pytest has at least pdb (not ipdb) support. Just invoke it with
pytest --pdb
and it drops into pdb with the first failure.
When running tests with py.test there is a --pdb option to enter pdb on failure.
Is there a similar way to enter the debugger when running the same test from within PyCharm?
There is a py.test plugin, pytest-pycharm, that will halt the PyCharm debugger when a test emits an uncaught exception.
Follow the steps below to setup the correct run configuration:
Run > Edit Configurations...
Click the '+' button to add a new configuration.
Name the configuration and specify the configuration parameters below:
Script: The path to your py.test executable (find by typing which py.test)
Script Parameters: This parameter is your test case followed by the --pdb option (ie /Users/Johan/projects/misc/testing.py --pdb)
After setting up the configuration, you can run the test case from within PyCharm. If a test case fails, your PyCharm run window will show the pdb prompt.
I am using py.test for unit testing my python program. I wish to debug my test code with the python debugger the normal way (by which I mean pdb.set_trace() in the code) but I can't make it work.
Putting pdb.set_trace() in the code doesn't work (raises IOError: reading from stdin while output is captured). I have also tried running py.test with the option --pdb but that doesn't seem to do the trick if I want to explore what happens before my assertion. It breaks when an assertion fails, and moving on from that line means terminating the program.
Does anyone know a way to get debugging, or is debugging and py.test just not meant to be together?
it's real simple: put an assert 0 where you want to start debugging in your code and run your tests with:
py.test --pdb
done :)
Alternatively, if you are using pytest-2.0.1 or above, there also is the pytest.set_trace() helper which you can put anywhere in your test code. Here are the docs. It will take care to internally disable capturing before sending you to the pdb debugger command-line.
I found that I can run py.test with capture disabled, then use pdb.set_trace() as usual.
> py.test --capture=no
============================= test session starts ==============================
platform linux2 -- Python 2.5.2 -- pytest-1.3.3
test path 1: project/lib/test/test_facet.py
project/lib/test/test_facet.py ...> /home/jaraco/projects/project/lib/functions.py(158)do_something()
-> code_about_to_run('')
(Pdb)
The easiest way is using the py.test mechanism to create breakpoint
http://pytest.org/latest/usage.html#setting-a-breakpoint-aka-set-trace
import pytest
def test_function():
...
pytest.set_trace() # invoke PDB debugger and tracing
Or if you want pytest's debugger as a one-liner, change your import pdb; pdb.set_trace() into import pytest; pytest.set_trace()
Similar to Peter Lyon's answer, but with the exact code you need for pytest, you can add the following to the bottom of your pytest module (my_test_module.py) :
if __name__ == "__main__":
pytest.main(["my_test_module.py", "-s"])
Then you can invoke the debugger from the command line:
pdb3 my_test_module.py
Boom. You're in the debugger and able to enter debugger commands. This method leaves your test code un-littered with set_trace() calls and will run inside pytest 'normally'.
Simply use: pytest --trace test_your_test.py.
This will invoke the Python debugger at the start of the test
I'm not familiar with py.test, but for unittest, you do the following. Maybe py.test is similar:
In your test module (mytestmodule.py):
if __name__ == "__main__":
unittest.main(module="mytestmodule")
Then run the test with
python -m pdb mytestmodule.py
You will get an interactive pdb shell.
Looking at the docs, it looks like py.test has a --pdb command line option:
https://docs.pytest.org/en/7.2.x/reference/reference.html#command-line-flags
Add and remove breakpoints without editing source files
Although you can add breakpoints by adding breakpoint() or set_trace() statements to your code, there are two issues with this approach:
Firstly, once you have started running your code, there is no way to remove your breakpoint. I often find that once I start running my code and reach an initial breakpoint, I want to place another one and remove the initial breakpoint. After breakpoint() drops me into the debugger I can add additional breakpoints, but I can't remove the initial one. Although this can be mitigated somewhat by putting the initial breakpoint statement higher up, if you have parametrised tests then even that is limited. I may find myself repeating cont very often.
Secondly, it requires changes to the source code. You need to remember to remove all breakpoint() commands before committing any code to version control, you have to remove them before switching branches, etc. I sometimes find I want to use the debugger to compare test runs between two branches, and having to edit the source code to add a breakpoint every time makes that a considerably slower and more error-prone exercise. I may even want to add a breakpoint in a library I'm calling, in which case the file I'm editing may not even me in my git repository but somewhere deep in my conda environment, increasing the risk of forgetting to remove it. Editing files to add break points is, in my humble opinion, ugly.
To add and remove breakpoints interactively without editing any source files, you can evoke pytest as follows (in the bash shell):
python -mipdb $(type -p pytest) -s test_fileset.py
The -s flag is crucial here, because it stops pytest from messing with stdin and stdout, and when running inside the debugger, pytest will fail to mess with stdin and stdout and everything will go wrong. The exact calling syntax will be different for different shells.