I have a collection of tests under one file test_file.py. I can run it normally from the console like this:
python -m unittest test_file
This outputs a small traceback when a test case fails. So what I need to do exactly is.
Run peridically the tests, let's say on crontab (I know how to do this)
send an email report after every run, in order to do this I need to know if all tests went out alright, and in case some of them failed, which ones failed and what the error was, just like the normal pyunit output.
As I said above, I know how to do the cron part and I know how to run the tests, but what do I need or what can I do to accomplish item 2 ?
Maybe a script that manually runs every test and collects the results and then send the email ?
Thank you very much !
if you intend on building out tests in the future, you should consider Jenkins. http://jenkins-ci.org/content/about-jenkins-ci . it can run your tests on a CRON, report results (with an xUnit plugin) per build over time, and conditionally send out an email based on the test results.
Related
I have a single unit test file, and I can run tests within Pycharm. Fine. However, it seems that Pycharm skips tests based on some criteria regarding code change. Something like, if the code in a test method hasn't changed, and/or if the code that a particular test method is testing hasn't changed, it won't run the test. This has caused a lot of pain, and has let a bug fall through that wasn't caught until much later on. So my question is, how to stop Pycharm from automatically skipping tests and force it to run all tests?
Why is skipping tests the default behaviour anyway? This seems absolutely outrageous to me, but please correct me if I'm wrong.
EDIT
Oops my bad. I ran the tests by pressing the keyboard shortcut control-shift-R on my Mac OS, which normally runs a Python script, but it doesn't actually run the whole test file, and only runs a single test (where the caret is) instead. This (keyboard shortcut having different behaviours) is a little bit misleading in my opinion, but regardless, my description of the problem is wrong and I was wrong. Sorry!
Make sure all your test methods staring with a test_.
Usually I have 2 Tests configurations: All and Current.
tests folder is located in the root of PyCharm project
I use python path to specify certain test to run.
You can easily copy the reference to certain test
I am currently developing some tests using python py.test / unittest that, via subprocess, invoke another python application (so that I can exercise the command line options, and confirm that the tool is installed correctly).
I would like to be able to run the tests in such a way that I can get a view of the code coverage metrics (using coverage.py) for the target application using pytest_cov. By default this does not work as the code coverage instrumentation does not apply to code invoked with subprocess.
Code Coverage of the code does work if I update the tests to directly invoke the entry class for the target application (rather than running via the command line).
Ideally I want to have a single set of code which can be run in two ways:
If code coverage monitoring is not enabled then use command line
Otherwise execute the main class of the target application.
Which leads to my question(s):
Is it possible for a python unit test to determine if it is being run with code coverage enabled?
Otherwise: is there any easy way to pass a command line flag from the pytest invocation that can be used to set the mode within the code.
Coverage.py has a facility to automatically measure coverage in sub-processes that are spawned: http://coverage.readthedocs.io/en/latest/subprocess.html
Coverage sets three environment flags when running tests: COV_CORE_SOURCE, COV_CORE_CONFIG and COV_CORE_DATAFILE.
So you can use a simple if-statement to verify whether the current test is being run with coverage enabled:
import os
if "COV_CORE_SOURCE" in os.environ:
# do what yo need to do when Coverage is enabled
Context
I have a python application that I'm unit testing. Half the application is working and I have a very high test accuracy.
The application requires one-time user input for installation purposes.
This means that, if you run the code, there has to be interaction with a user.
Problem
Coverage is a Python plugin for coverage reports. I use coverage with this command:
coverage run application.py
Coverage runs my application, goes through my tests, and delivers a coverage report.
The problem is that the command to run those tests, executes my application and I have to provide input. That's not that big of a deal, but I cannot do that on my CI server using Jenkins (or can I?).
Question
I want to run the coverage tool without user input. In my tests, the input function is mocked out. Running all my tests without coverage works fine. How can I prevent coverage from requiring user input?
You should probably have 2 different code paths, one for running the tests, and one for running the app:
coverage run tests.py
with tests.py importing application.py, mocking methods as necessary, then running the actual application.
Or you could allow user input via command line arguments:
coverage run application.py --user=input --other="etc."
Finally, if there truly are portions of your app that cannot be tested or reasonably mocked (it happens, say you're calling out into a third party exception tracking library/service that you can't load in your tests), you can instruct coverage to ignore those lines for the purposes of computing coverage, by adding # pragma: no cover at the end of the instruction that you won't be fully testing:
my = "code"
goes = "here"
if debug: # pragma: no cover
call_untestable(code=True)
this_portion(ignored_for_coverage=True)
covered_code = "yes, again!"
See more here:
http://coverage.readthedocs.io/en/coverage-4.2/excluding.html
I have 60 functional tests in one file. I wrote them in Notepad++ and used py.test as the test framework. Today I decided to swap Notepad++ with PyCharm. I opened my file of functional tests in PyCharm and ran the tests from PyCharm, as you can see in the picture:
Now, after confirming that I could run all of the tests, I tried to run an individual test, for example test_login_with_extantUser_using_email. Logically, I right-clicked on the test, expecting a "run test" button or something similar to appear. But no such thing appeared. In fact, it appears that there is no way to run an individual test by simply right-clicking on it.
So my question is, how can I run an individual test? Must I set up a configuration for each one in the Edit Configurations menu? That would take a very long time, considering that I have 60 tests.
I have found the problem. It's a bug in Pycharm that needs to be fixed. I will explain the bug.
Right-clicking an individual test will not display the option to run the test if the test is not a member of a class that inherits from unittest.TestCase. This is true even if you are not using unittest, as in my case, in which I am using py.test.
When I made my py.test test classes inherit from unittest.TestCase, I got the option to run tests individually when I right-clicked on a test.
I have reported the bug to Pycharm. Time will tell if they fix it.
https://youtrack.jetbrains.com/issue/PY-26754
Base on pt.test doc you can specify particular test_case for running with -k (which means keywords obviously). In your case it should be like pytest -k "test_login_with_extantUser_using_email" if you running in CLI.
If you want to run particular test with Pycharm, you have to fill 'Keywords' field in Run/Debug Configurations.
Best Regards.
In a Django project, it is possible to create unit-tests to verify what we had done so far. The principle is simple. We have to execute the command python3 manage.py test in the shell. When an error is detected in the program, the shell will display it and stop the process. However, the procedure has a little gap. If we have several errors, we have to correct it and restart the whole process. This process could take several minutes which depends of our program. Is there a manner to restart the process where the error has been detected instead of restart the whole procedure?
EDIT :
In fact, another problem I have is to retain the databases instead of recreate it. How could I do such thing?
If you want to automatically run only failing tests you need to use a third party testing driver like Nose or create your own. But it's not worth it because ...
You can specify particular tests to run by supplying any number of
“test labels” to ./manage.py test. Each test label can be a full
Python dotted path to a package, module, TestCase subclass, or test
method. For instance:
# Run just one test method
$ ./manage.py test animals.tests.AnimalTestCase.test_animals_can_speak
Source: https://docs.djangoproject.com/en/1.10/topics/testing/overview/
This approach can be used to re run only the ones that have failed.
Please note that third party test runners will probably recreate the database every time you run the test - even for only the failing test. On the other hand the django default test runner has the --keep option which allows the database to be reused. For more details see: https://stackoverflow.com/a/37100979/267540