I'm using Pytest to test my code and I'm running into a little, but an infuriating issue.
One of the first things my program does is checking if there are any setting files available. If there aren't any it throws an error and calls exit(). This works well during normal runtime but messes with Pytest.
The solution I came up with is to simply create a temporary settings file for the duration of the tests, by copying the template settings file. I've already written and successfully tested the code to achieve that.
The problem I'm running into is that I can't find a Pytest hook that truly fires before everything else. This results in the program throwing the error before Pytest tries to create a temporary settings file. Which results in Pytest failing before it can perform any tests.
Does anyone know of a way to fire a function before Pytest does or loads anything? Preferably within Pytest itself.
Some code context:
Throwing the error and exit
This snippet runs on the import of the settings module.
if len(cycles) == 0:
log.error("No setting files found. Please create a file " +
"in the `settings` folder using the Template.py.")
exit()
Creating the temporary settings file
This code should be the very first thing Pytest runs.
def pytest_sessionstart(session):
""" Create a test settings file """
folderPath = os.path.dirname(os.path.abspath(__file__))
folderPath = os.path.split(folderPath)[0] + "/settings"
srcfile = folderPath + '/Template.py'
dstfile = folderPath + '/test.py'
shutil.copy(srcfile, dstfile)
Removing the temporary settings file
This code should be one of the last things Pytest runs.
def pytest_sessionfinish(session, exitstatus):
""" Delete the test settings file """
folderPath = os.path.dirname(os.path.abspath(__file__))
folderPath = os.path.split(folderPath)[0] + "/settings"
os.remove(folderPath + "/test.py")
Pytest output
With the call to exit() disabled, so you can see the execution order.
Lakitna$ pytest -v -s
No setting files found. Please create a file in the `settings` folder using the Template.py.
TEMPORARY SETTINGS FILE CREATED
========================= test session starts ==========================
platform darwin -- Python 3.6.4, pytest-3.3.1, py-1.5.2, pluggy-0.6.0 -- /usr/local/opt/python3/bin/python3.6
cachedir: .cache
rootdir: /Users/lakitna/Documents/Github/Onaeri-tradfri/Onaeri, inifile:
collected 24 items
Add conftest.py to your project with below code and add your code to method pytest_configure
def pytest_configure(config):
pass # your code goes here
# other config
First, pytest tries to make all imports as relative as possible: this means that if you keep your tests in package/tests, then even your conftest will be imported as package.test.conftest. To do that import, package.__init__ will be called (unfortunately for you).
In order to run run some configuration before, the tests need to be moved out of the package tree. I ran into just such a problem recently.
As shown in a pytest issue thread, pytest_configure is actually called before pytest_sessionstart, but both occur before test collection, so either should be fine.
# contents of tests/conftest
import os
from pathlib import Path
import shutil
import pytest # if necessary
# cannot import package here (and likely not in `tests.__init__` either)
folderPath = Path.cwd()
folderPath = folderpath / 'package' / 'settings'
srcfile = folderPath / 'Template.py'
dstfile = folderPath / 'test.py'
def pytest_configure(config):
""" Create a test settings file """
# can use `config` to use command line options
shutil.copy(srcfile, dstfile)
def pytest_unconfigure(config):
""" Delete the test settings file """
os.remove(dstfile)
This should work if you have your folders structured like this:
root_folder
├── package
│ ├── settings
│ │ └── Template.py
│ ├── __init__.py
│ └── module1.py
└── tests
├── __init__.py # Does not import package
├── conftest.py # Does not import package
└── test_module1.py # Can import package
pytest can then be invoked in the normal way: pytest -flags (or pytest tests -flags) when working from root_folder.
If you want to set up your settings file in different ways depending on custom command line flags to pytest, you can do that as well in conftest.py. They can then be accessed in pytest_configure as config.getoption(...).
Note: this syntax assumes Python 3.6+ with the use of Paths as arguments to os.remove and shutil.copy.
I've managed to find a solution.
After some additional research, I found out that Pytest preloads all modules. This means that you can never run code before a module import unless you can find a hook before the collection phase. There is no such hook as far as I know. I really wanted to make this work within Pytest, but it seems to be impossible.
Instead, I created a __main__.py file in my test folder with the following content:
import pytest
import os
import shutil
"""
Create a setting file for the test procedure
"""
folderPath = os.path.dirname(os.path.abspath(__file__))
folderPath = os.path.split(folderPath)[0] + "/settings"
srcfile = folderPath + '/Template.py'
dstfile = folderPath + '/test.py'
shutil.copy(srcfile, dstfile)
"""
Actually run pytest
"""
pytest.main()
"""
Remove the test settings file
"""
os.remove(dstfile)
This code creates a temporary settings file, starts Pytest, and then removes the temporary file again.
I can now run the test procedure as follows:
$ python3 test
Pytest flags still work as normal. For example, if you want more verbose output you can do the following:
$ python3 test -v
This deviates from what you are looking for, but I would question your strategy. I suspect your code is not test friendly. The available hooks should suffice for setting up test pre-conditions like that. That is if your tests need some file, somehow setup that file first in setup, delete them with teardown, or use fixtures as per the pytest docs.
If you don't actually need the file for the tests to run, except that the code tests its existence, you can fake its existence with mocks.
But you have code running on module import that is testing (without actually being test code) a pre-condition and with user interaction.
And I think you shouldn't do this.
Remove the execution of that code running with import (make it a function or class). And invoke it in some other initialising place of your code.
Setup the file (with pytest hooks). Test code that uses the settings file.
Teardown the file (with pytest hooks).
You can in alternative completely fake the file by overriding the data read with fake data, using some data struture and functions to read/write,
simulating the file.
Test the code that tests for the settings file existence (use mocks).
Test your other code methods.
Do not mix things. Try to write methods that do one thing. Test one thing at a time. Test your business code. User interaction is another thing. If you also want to test UI, don't do it at the same time (I mean in the same code).
To run some code before and after the actual code of your test, you can use pytest fixtures:
import pytest
#pytest.fixture
def context():
"""What to do before/after the test."""
print("Entering !")
yield
print("Exiting !")
def test_stuff(context):
"""The actual code of your test"""
print("Running the test")
This displays:
Entering !
Running the test
Exiting !
To use the fixture, just pass a parameter with the same name as input of the test function. The yield instruction acts like a separator between what's done before and after the test.
Related
Error: While importing "wsgi-contract-test", an ImportError was raised:
Traceback (most recent call last):
File "/Users/karl/Development/tral/bin/tral-env/lib/python3.9/site-packages/flask/cli.py", line 236, in locate_app
__import__(module_name)
File "/Users/karl/Development/tral/test/contract/inject/wsgi-contract-test.py", line 8, in <module>
from . import (
ImportError: attempted relative import with no known parent package
ERROR (run-tral): trap on error (rc=2) near line 121
make: *** [component.mk:114: tral-local-run-api-contract-test] Error 2
wsgi-contract-test.py:
from ...libs.tral import app, setup_from_env
from ...libs.tral import routes
I have the regular source files under the libs/tral directory, however the entry file for this test is located under test/contract/inject. I do NOT want to move this test file into libs since this file should be nowhere near production code as it is a rather hazardous file security wise.
In node.js this would of worked fine but there seems to be something with python imports I'm not grasping?
Since tests don't belong inside the src tree, there are basically two approaches. If you are testing a library you will frequently install it (in a virtualenv) and then just import it exactly the same way you would anywhere else. Frequently you also do this with a fresh install for the test: this has the great advantage that your tests mirror real setups, and helps to catch bugs like forgetting to commit/include files (guilty!) and only noticing once you've deployed...
The other option is to modify sys.path so that your test is effectively in the same place as your production code:
# /test/contract/inject.py
from pathlib import Path
import sys
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
Since sys.path uses strs you may prefer to use os.path, like every single other example on SO. Personally I'm addicted to pathlib. Here:
__file__ is the absolute path of the current file
Path(str) wraps it in a pathlib.Path object
.parent gets you up the tree, and
str() casts back to the format sys.path expects, or it won't work.
Using a test runner
Normally we use a test runner for tests. But these tests need a running instance! No problem:
# tests/conftest.py
import pytest
from sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent))
# now pytest can find the src
from app import get_instance # dummy
#pytest.fixture
def instance():
instance = get_instance(some_param)
# maybe
instance.do_some_setup()
yield instance
# maybe
instance.do_some_cleanup()
If you don't need to do any cleanup, you can just return the instance rather than yielding it. Then in another file (for neatness) you write tests like this:
# tests/test_instance.py
def test_instance(instance): # note how we requested the fixture
assert instance.some_method()
And you run your tests with pytest:
pytest tests/
Pytest will run the code in conftest.py, discover all test fns starting with test in files whose names start with test, run the tests (supplying all the fixtures you have defined) and report.
Fixture Lifetime
Sometimes spinning up a fixture can be expensive. See the docs on fixture scope for telling pytest to keep the fixture around and supply it to multiple tests.
Using a runner vs. just running a script
You don't have to use a runner. But they do have many advantages:
decent metrics e.g. code/branch coverage
everyone uses them
parallel testing against different envs (e.g. python versions)
tests tend to be a lot neater
I took for granted that Python was able to handle simple relative paths; not the case. Instead I just added the path to the packages I wanted to include in the PYTHONPATH variable and walla, it found everything.
export PYTHONPATH="${PYTHONPATH}:$(pwd)/libs"
Then running it from the root project directory.
I had to change the code to the following though:
from tral import app, setup_from_env
from tral import routes
I have the following pytest directory structure:
system_tests/
├── conftest
├── pytest.ini
│
├── suite_1/
│ └── test_A.py
│
└── suite_2/
└── sub_suite_2a/
└── test_B.py
When each test method runs, a number of third-party libraries/processes generate artifacts in the current working directory.
When pytest is executed from the sub_suite folder (using CLI or IDE "play" button), the files are generated in the sub_suite folder, where I want them to be.
However, when pytest is run from the system_tests folder to run all tests, all artifacts are created in the system_tests folder, which is not what I want.
Is there an easy way to force pytest to always use the test class folder as the working directory so I get the same results regardless of how or where I run a test from?
EDIT: Improved Solution
Using monkeypatch as suggested by #Kound removes the boilerplate code to restore the cwd. You can also enable autouse to automatically apply this fixture to all test functions. Add the following fixture to conftest.py to change the cwd for all tests:
#pytest.fixture(autouse=True)
def change_test_dir(request, monkeypatch):
monkeypatch.chdir(request.fspath.dirname)
Any processes that are kicked off by the test will use the test case folder as their working directory and copy their logs, outputs, etc. there, regardless of where the test suite was executed.
Original Solution
The following function-level fixture will change to the test case directory, run the test (yield), then change back to the calling directory to avoid side-effects, as suggested by #hi2meuk:
#pytest.fixture
def change_test_dir(request):
os.chdir(request.fspath.dirname)
yield
os.chdir(request.config.invocation_dir)
request is a built-in pytest fixture
fspath is the LocalPath to the test module being executed
dirname is the directory of the test module
request.config.invocationdir - the folder from which pytest was executed
request.config.rootdir - pytest root, doesn't change based on where you run pytest. Not used here, but could be useful.
A different and, IMHO more robust approach: always reference your files by the complete path.
__file__ is an automatically declared Python variable that is the name of the current module. So in your test_B.py file, it would have the value: system_tests/suite_2/sub_suite_2a/test_B.py. Just get the parent and choose where to write your files.
from pathlib import Path
test_data_dir = Path(__file__).parent / "test_data"
Now you have all of them in the same place and can tell your version control system to ignore them.
If the code is inside a library, better use an absolute path:
import os
from pathlib import Path
test_data_dir = Path(__file__).parent.absolute() / "test_data"
Instead of creating a fixture for each directory like suggested by #DV82XL you can simply use monkeypatch to achieve the same:
import pytest
from pathlib import Path
#pytest.fixture
def base_path() -> Path:
"""Get the current folder of the test"""
return Path(__file__).parent
def test_something(base_path: Path, monkeypatch: pytest.MonkeyPatch):
monkeypatch.chdir(base_path / "data" )
# Do something in the data folder
Many options open to you to achieve this. Here are a few.
1.
Write a pytest fixture to check if the current working directory is equal to the desired working directory, and if not, then move all the artifact files to the desired directory. If the artifacts you are generating are all the same type of file (e.g. *.jpg, *.png, *.gif) and you just want them to be in a different directory, then this may suffice. Something like this could work
from pathlib import Path
import shutil
#pytest.fixture
def cleanup_artifacts():
yield None
cwd = Path.cwd()
desired_dir = Path.home() / 'system-tests' / 'suite-2' / 'sub_suite_2a'
if cwd != desired_dir:
for f in cwd.glob('*.jpg'):
shutil.move(f, desired_dir)
And then you can add this fixture to your tests as needed.
2.
You can configure the pytest rootdir to be the desired directory, since pytest uses the rootdir to store project/testrun specific info.
When you run pytest, run it as
pytest --rootdir=desired_path
See here for more info: https://docs.pytest.org/en/latest/customize.html#initialization-determining-rootdir-and-inifile
If both don't work for you, tell more about what your requirements are. Surely this can be done with pytest.
I know this violates any best practices which require/assume proper packaging of Python production code: In some cases it might be helpful to beeing able to define production and test code in the same file (e.g. in case of simple scripts). How can I run all or specific tests in a file with pytest then?
EDIT - The solution for my particular use case:
file structure in <root>:
pytest.ini
scrip_with_tests.py
content of pytest.ini:
[pytest]
python_files = script_with_tests.py
content of script_with_tests.py:
import pytest # this is not required, works without as well
def test_always_pass():
pass
if __name__ == "__main__":
main()
pytest invocation in <root>:
pytest script_with_tests.py
As stated in the Customizing test collection section:
You can easily instruct pytest to discover tests from every Python file:
# content of pytest.ini
[pytest]
python_files = *.py
However, many projects will have a setup.py which they don’t want to be imported. Moreover, there may files only importable by a specific python version. For such cases you can dynamically define files to be ignored by listing them in a conftest.py file:
# content of conftest.py
import sys
collect_ignore = ["setup.py"]
if sys.version_info[0] > 2:
collect_ignore.append("pkg/module_py2.py")
I've got a large conftest.py file that I wish to split into smaller parts, for two reasons:
The file is very large (~1000 lines, including documentation)
Some of the fixtures depend on other fixtures, and I have no reason to expose those other fixtures as part of the conftest "API" when users look for relevant fixtures
I am not aware of any mechanism provided by pytest to resolve conftest files in multiple locations within the same folder, so I contrived one, below:
import sys
import os
sys.path.append(os.path.dirname(__file__))
from _conftest_private_part_1 import *
from _conftest_private_part_2 import *
from _conftest_private_part_3 import *
#pytest.fixture
def a_fixture_that_is_part_of_the_public_conftest_api():
pass
This works for my needs, but I do wonder if there is a better way.
You can put your stuff in other modules and reference them using a pytest_plugins variable in your conftest.py:
pytest_plugins = ['module1', 'module2']
This will also work if your conftest.py has hooks on them.
You shouldn't need any fancy magic for that. py.test automatically adds the path of the current test file to sys.path, as well as all parent paths up to the directory it was targeted at.
Because of that, you don't even need to put that shared code into a conftest.py. You can just put into plain modules or packages and then import it (if you want to share fixtures, those have to be in a conftest.py).
Also, there is this note about importing from conftest.py in the documentation:
If you have conftest.py files which do not reside in a python package
directory (i.e. one containing an __init__.py) then “import conftest”
can be ambiguous because there might be other conftest.py files as
well on your PYTHONPATH or sys.path. It is thus good practise for
projects to either put conftest.py under a package scope or to never
import anything from a conftest.py file.
Alternativelly, you could try:
In my_fixtures.py:
#pytest.fixture
def some_fixture():
yield
In conftest.py:
import my_fixtures
# Adding the fixture to attributes of `conftest` will register it
some_fixture = my_fixtures.some_fixture
It seems that pytest detect fixture by iterating over conftest attributes and checking for some attr._pytestfixturefunction added by #pytest.fixture.
So as long as conftest.py contains fixture attributes, it doesn't really matter which file is the fixture defined:
This works for me and seems easier/clearer:
Top level tests/conftest.py (example of re-usable print debug of requests.Response):
import pytest
import requests
from requests_toolbelt.utils import dump
#pytest.fixture(scope="session")
def print_response(response: requests.Response):
data = dump.dump_all(response)
print("========================")
print(data.decode('utf-8'))
print("========================")
print("response.url = {}".format(response.url))
print("response.request = {}".format(response.request))
print("response.status_code = {}".format(response.status_code))
print("response.headers['content-type'] = {}".format(response.headers['content-type']))
print("response.encoding = {}".format(response.encoding))
try:
print("response.json = {}".format(response.json()))
except Exception:
print("response.text = {}".format(response.text))
print("response.end")
From lower level conftest import the higher level conftest code - e.g., tests/package1/conftest.py:
from tests.conftest import *
Then, in your lower level tests within tests/package1/test_*.py, you simply import via:
from tests.package1 import conftest
And then you have the merged configtests from one conftest available. Repeat this pattern for your other lower level detailed/modular conftest.py files throughout the tests hierarchy.
Few times while browsing tests dir in various Django apps I stumbled across models.py and settings.py files (in django-tagging for example).
But there's no code to be found that syncs test models or applies custom test settings - but tests make use of them just as if django would auto-magically load them. However if I try to run django-tagging's tests: manage.py test tagging, it doesn't do even a single test.
This is exactly what I need right now to test my app, but don't really know how.
So, how does it work?
If you want to run the tests in django-tagging, you can try:
django-admin.py test --settings=tagging.tests.settings
Basically, it uses doctests which are in the tests.py file inside the tests package/directory. The tests use the settings file in that same directory (and specified in the command line to django-admin). For more information see the django documentation on writing doctests.
You mean, "How do I write unit tests in Django?" Check the documentation on testing.
When I've done it, I wrote unit tests in a test/ subdirectory. Make sure the directory has an empty __init__.py file. You may also need a models.py file. Add unit tests that derive from unittest.TestCase (in module unittest). Add the module 'xxxx.test' to your INSTALLED_APPS in settings.py (where 'xxxx' is the base name of your application).
Here's some sample code of mine to get you started:
#!/usr/bin/env python
# http://docs.djangoproject.com/en/dev/topics/testing/
from sys import stderr
import unittest
from django.test.client import Client
from expenses.etl.loader import load_all, load_init
class TestCase(unittest.TestCase):
def setUp(self):
print "setUp"
def testLoading(self):
print "Calling load_init()"
load_init()
print "Calling load_all()"
load_all()
print "Done"
if __name__ == '__main__':
unittest.main()
If you mean, "How do I get data loaded into my unit tests?", then use fixtures, described on the same documentation page.