I have a python app that I’m attempting to unit test. The code is sat in a src directory the tests in a test directory.
The problem comes when trying to test the class containing my main method. This class sits in the src directory and has references to classes in other modules that all sit in the src directory. In order to execute my app I have the references in this format from <module> import <class>
When I come to test this main class I write a test class and place it in the test directory. Running the test I find I have problems with the imports in the main class. To resolve I have to change the imports to be from from src.<module> import <class> Having done this my tests now pass but the app itself fails.
How can I resolve the issue so that my imports are valid for both my unit tests and normal execution?
It may fail due to the way import works.
I would recommend always using absolute imports. This means, that when creating a package, all imports within that package import all the way from the top level down: from package.module import thing. And more importantly, installing your package - not as a normal Python package, but with a symbolic link allows you to edit your source code while it still is considered an installed package. This way you can position your tests and scripts however you want them and the thing will always work.
Installing your package is done with pip install -e . from root. See a packaging tutorial for an example of packaging if you're not familiar with it.
An example package structure might be:
root/
setup.py
MANIFEST.in
...
package/
__init__.py
app.py
tests.py
module/
__init__.py
component.py
module_test.py
# Or have a tests folder under root
tests/
test_module.py
tests.py
And for example in app.py, and everywhere really:
from package.module.component import Class
from package.module import thing # that was declared in module/__init__.py
So to your case, if I understood it correctly, you'd have:
root/
setup.py
MANIFEST.in
...
src/
__init__.py
app.py
module.py
tests/
test_module.py
tests.py
Related
I have a Python project in which I have code organized as follows:
/package
LICENSE
README.md
setup.py
/project
/mod_1
mod_1.py
__init__.py
/tests
test_mod_1.py
/mod_2
mod_2.py
__init__.py
/tests
test_mod_2.py
object_factory.py
__init__.py
The object_factory.py file contains a class that is used to build objects defined in mod_1.py and mod_2.py.
Within the tests, each test file imports the class from object_factory.py. When I run pytest from the /project directory, it complains that the object_factory module does not exist. If, however, I remove __init__.py from each of the mod_# directories, it works fine.
I've read many blog posts and Python help posts describing the intent of __init__.py as well as pytest, but I cannot seem to sort out this one detail.
1. __init__.py files
I've read many blog posts and Python help posts describing the intent of init.py as well as pytest, but I cannot seem to sort out this one detail.
Answering your question: __init__.py files are used to mark your directory as a Python package. This means that Python will look for submodules inside that directory, so you can import them.
Alongside, pytest requires you to write tests for your package in a separate packages. It means that you should consider putting __init__.py in your tests directory.
2. Project structure
Talking about project structure, pytest provides two basic test layouts in their Good integration practices
The first one is Tests outside application code
pyproject.toml
setup.cfg
mypkg/
__init__.py
app.py
view.py
tests/
test_app.py
test_view.py
...
as documentation says, this approach has a bunch of benefits:
Your tests can run against an installed version after executing pip install ..
Your tests can run against the local copy with an editable install after executing pip install --editable ..
The second one is Tests as part of application code
pyproject.toml
setup.cfg
mypkg/
__init__.py
app.py
view.py
test/
__init__.py
test_app.py
test_view.py
...
This is actually your case.
Turning back to your project structure, here is a couple of things I want to highlight:
As I've said before, consider making your test directories packages.
Take a look at PEP 423 - Avoiding deep nesting. You probably don't want to complicate your package, therefore, don’t put your package modules in subdirectories unless you really need to.
Taking into account all the above, I would recommend you change your structure a little bit, so it looks like the following:
/package
LICENSE
README.md
setup.py
/project
mod_1.py
mod_2.py
/tests
__init__.py
test_mod_1.py
test_mod_2.py
object_factory.py
__init__.py
In this case, you should run you tests with the command
pytest --pyargs project
Note that you have to import your modules into object_factory.py using from project import test_mod_X
However, you probably don’t even want to distribute your test along with your application.
If I'm right, you can also use the first approach:
/package
LICENSE
README.md
setup.py
/project
__init__.py
mod_1.py
mod_2.py
object_factory.py
/tests
__init__.py
test_mod_1.py
test_mod_2.py
and run your test with pytest. It seems more readable and straightforward to me.
You may also find it useful to look at Invoking pytest versus python -m pytest
One more thing
There is one more thing I would like to add. It's something that isn't directly related to your problem but can cause confusion and, as a result, other problems later on when your code is more complex.
You may have confused the names project and package.
The project is a folder in which all your files are located - README, LICENSE, setup.py, etc.
The package is the directory where __init__.py, your code, and modules reside.
With that in mind, swap these names around to make your code more readable for yourself and others.
For Python 3.
I just started learning Python. I have PHP and Ruby background.
Currently very confused with modules, __init__ and python -m.
At the moment I have the following:
modules/practice.py
tests/test_practice.py
In practice.py
class First:
def attempt(self):
return 'attempted'
In test_practice.py
from modules.practice import First
class TestMain:
def test_attempt(self):
first = First()
attempted = first.attempt()
assert attempted is 'attempted'
When I run pytest I get an error ModuleNotFoundError: No module named 'modules
When I run python -m pytest test is green.
However, if I add __init__.py files in modules and tests, both are green.
After trying to find out answers on my own I confess I am not sure I am getting it.
Why pytest does not work without __init__?
When working on a project, when people assume python -m will be used and when people add the __init__.py files instead?
Do not add an __init__.py file into the tests directory.
You should add an __init__.py file into the modules directory.
Ideally it should be possible to run the test suite with either pytest or python -m pytest. For this to work, the parent directory of modules needs to be present on sys.path. Usually you would do this by writing a setup.py file for the package and installing your code in "editable" mode. The tests run against the installed code linked in site-packages.
If you don't want to write a setup.py file at this stage, you can either export an environment variable:
export PYTHONPATH=/path/to/parent_of_modules
Or you can inject to sys.path directly from conftest.py file, which gets imported first:
.
├── modules
│ ├── __init__.py
│ └── practice.py
└── tests
├── conftest.py
└── test_practice.py
Example:
# conftest.py
import os
import sys
here = os.path.dirname(os.path.abspath(__file__))
sys.path.append(os.path.dirname(here))
I am new at Python and coming from Java background.
Suppose I am developing a Python project with package hello:
hello_python/
hello/
hello.py
__init__.py
test/
test_hello1.py
test_hello2.py
I believe the project structure is correct.
Suppose hello.py contains function do_hello() I want to use in tests. How to import do_hello in tests test_hello1.py and test_hello2.py ?
You've 2 small issues here. Firstly, you're running your test command from the wrong directory, and secondly you've not quite structured your project right.
Usually when I'm developing a python project I try to keep everything focussed around the project's root, which would be hello_python/ in your case. Python has the current working directory on its load path by default, so if you've got a project like this:
hello_python/
hello/
hello.py
__init__.py
test/
test_hello1.py
test_hello2.py
# hello/hello.py
def do_hello():
return 'hello'
# test/test_hello.py
import unittest2
from hello.hello import do_hello
class HelloTest(unittest2.TestCase):
def test_hello(self):
self.assertEqual(do_hello(), 'hello')
if __name__ == '__main__':
unittest2.main()
Second, test isn't a module right now, since you've missed the __init__.py in that directory. You should have a hierarchy that looks like this:
hello_python/
hello/
hello.py
__init__.py
test/
__init__.py # <= This is what you were missing
test_hello1.py
test_hello2.py
When I try that on my machine, running python -m unittest test.hello_test works fine for me.
You may find that this is still a bit cumbersome. I'd strongly recommend installing nose, which will let you simply invoke nosetests from your project's root to find and execute all your tests automagically - providing you've got correct modules with __init__.pys.
In my project, have a subdirectory containing a python application I'd like to unit-test.
The structure looks as follows:
my-project/
python-app/
sometool/
__init__.py
foo/
__init__.py
aaa.py
bar/
__init__.py
bbb.py
test/
sometool-tests/
foo-tests/
aaa_test.py
Now, aaa.py contains imports like import sometool.bar.bbb, which assumes the application's base directory is python-app, which is indeed the case in my build setup.
aaa_test.py obviously imports aaa for testing it.
However, when running nosetests ./python-app/sometool/test from the main project directory, imports fail because my-project is now the base directory for imports, i.e. sometool.bar.bbb is not found from there.
If I cd into python-app first and run nosetests ./sometool/test from there, everything works as expected.
But I would like to configure Visual Studio Code to run those tests using a shortcut, and those command always seem to be executed from the project root.
Is there a way to pass the "base directory" as argument to Nose?
You can use a context providing module in the nosetests environment:
my-project/
python-app/
...
test/
context.py
context.py:
import sys
import os
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
import python-app
Then you can import python-app in your tests import from the context module, e.g. in aaa_test.py:
from tests.context import python-app
from python-app.sometool import foo, bar
This should make nosetests always find the python app, no matter where it is being executed.
I have a package and a test package. According to advice from Where do the Python unit tests go?, the tests should be in a different directory. The project's directory tree is as follows:
project\
kernel\
__init__.py
file1.py
file2.py
tests\
__init__.py
test1.py
test2.py
test3.py
I would like to import the kernel package to the tests package, because that is where file1.py and file2.py are being tested. Also, I would like to use one import statement in the __init__.py instead of importing kernel again and again in each test.
I tried adding the following to the __init__.py file in tests and to test2.py , test2.py (together and separately), with no success (the first does no harm, the second gives a syntax error):
import kernel
import ../kernel
I'm using python2.6. From command line all the above happens. When I use Eclipse PyDev, everything magically works.
The relative imports you're using will only work if the "project" directory is a python package (i.e. it has an __init__.py file in it). Try that first and see if that works for you.
If the kernel directory is acting as the "package" that would be distributed then you could put the tests directory inside of that and do the relative imports that way. So it would look like this:
project/
kernel/
__init__.py
file1.py
file2.py
tests/
__init__.py
test1.py ...
And you would import the kernel modules from the tests directory either as:
from kernel import file1 # if it's installed in the python path/environment
Or:
from .. import file1
# 'import ..file1' might work, but I'm not sure that's syntactically correct