I am following the pytest "Get Started" guide, and I just can't make it work. It seems to be something very elementary, but I just cant find it. The problem resides in importing modules from other folders, and, although I am following the official documentation, I cant make it work
Following PyPa and the official documentation, I have assembled the following file structure:
(lpthw) ex47_pypa> tree
.
├── dist
│ ├── example_package_vcmota-0.0.1-py3-none-any.whl
│ └── example_package_vcmota-0.0.1.tar.gz
├── LICENSE
├── pyproject.toml
├── README.md
├── src
│ └── example_package_VCMota
│ ├── example.py
│ └── __init__.py
└── tests
└── test_example.py
4 directories, 8 files
(lpthw) ex47_pypa> cat src/example_package_VCMota/example.py
def add_one(number):
return number+1
(lpthw) ex47_pypa> cat tests/test_example.py
# import example
# import example_package_VCMota
# from src/example_package_VCMota import example
# from src import example_package_VCMota/example
# import src
# import importlib
# from src.example_package_VCMota.example import add_one
from ..src.example import add_one
def test_dif():
assert add_one(2) == 9
(lpthw) ex47_pypa>
That seems to me as an ok implementation of everything in pytest pages, except that it does not work:
(lpthw) ex47_pypa> pytest
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.10.8, pytest-7.1.3, pluggy-1.0.0
rootdir: /archive/MyBooks/LearnPython3TheHardWay/mypython/projects/ex47_pypa
collected 0 items / 1 error
====================================================================================================== ERRORS ======================================================================================================
______________________________________________________________________________________ ERROR collecting tests/test_example.py ______________________________________________________________________________________
ImportError while importing test module '/archive/MyBooks/LearnPython3TheHardWay/mypython/projects/ex47_pypa/tests/test_example.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_example.py:8: in <module>
from example_package_VCMota.example import add_one
E ModuleNotFoundError: No module named 'example_package_VCMota'
============================================================================================= short test summary info ==============================================================================================
ERROR tests/test_example.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================================================= 1 error in 0.05s =================================================================================================
(lpthw) ex47_pypa>
The error is due to how I call the function add_one in example.p, and, as you may see in the test_example.py, I have tried every imaginable way of calling this function.
The only thing I know for sure is that the error has nothing to do with pytest: I have assembled another file structure:
teste> tree
.
├── src
│ ├── example.py
│ └── __init__.py
└── tests
└── test_example.py
2 directories, 3 files
teste> cd tests/
tests> cat test_example.py
# import example
# import example_package_VCMota
# from src/example_package_VCMota import example
# from src import example_package_VCMota/example
# import src
# import importlib
# from src.example_package_VCMota.example import add_one
from ..src.example import add_one
def test_dif():
assert add_one(2) == 9
tests>
but it still does not work:
tests>python test_example.py
Traceback (most recent call last):
File "/archive/MyBooks/LearnPython3TheHardWay/mypython/projects/teste/tests/test_example.py", line 10, in <module>
from ..src.example import add_one
ImportError: attempted relative import with no known parent package
tests>
I am aware that there are multiple other ways of importing modules from other folders, as stated in this 11 years old post from stackoverflow itself, but the dot one is surely the simplest way, and, since it is clearly stated in the official documentation, it should work, but I cant make it work.
Thank you all for your attention.
EDIT:
Following Raphael's comments, I have started to suspect of errors/problems in my python installs, and therefore assembled the following file structure inside a virtual machine running Ubuntu LTS 20:
├── source
│ ├── example.py
│ └── init.py
└── tests
└── test_example.py
where I have tried multiple import commands in test_example.py, such as:
from .source.example import add_one
from ...source.example import add_one
and so on, and, again, none have worked. Since I am running Gentoo in my main machine, I suspect that this test indicates that whatever is the issue, it is certainly not with my python installs.
It's a common issue in Python that the tests for a package cannot find the package itself.
The main reason for the issues is your working directory. You cannot just cd into the tests folder and run the tests. In fact, you need to be one level above your project folder, so you need to be in ex47_pypa/.. and then run python -m ex47_pypa.tests.test_example.
This will give Python enough directory hierarchy to actually resolve the relative imports in your tests, so the from ..src.example can be resolved.
I've created an experimental, new import library: ultraimport
It gives you more control over your imports and lets you do file system based imports.
In your test_example.py your could then write:
import ultraimport
add_one = ultraimport('__dir__/../src/example_package_VCMota/example.py', 'add_one')
This will always work, no matter how you run your code or what's your current working directory or what's in your sys.path.
Related
I'm having trouble importing test helpers into test code.
The project's python source code and test code directories are separate, but their structure parallels one another. In the test directory, I have some helper test code that I want to import into the actual tests. I am setting PYTHONPATH to include both the main src dir and the test dir (details below).
Command line:
PYTHONPATH="$(pwd)/src/main/python:$(pwd)/src/test/python" poetry run python -m pytest --import-mode=importlib ${#} ./src/test/python
Expectation:
Importing a test helper module from tests would work.
Actual:
_ ERROR collecting src/test/python/module_A/module_1/test_file.py _
ImportError while importing test module '/Users/<username>/git/src/test/python/module_A/module_1/test_file.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
src/test/python/module_A/module_1/test_file.py: in <module>
from module_A.module_1.test_helpers.helper_a import HelperA
E ModuleNotFoundError: No module named 'module_A.module_1.test_helpers'
Note: If I change the test directory structure to no longer parallel the src directory structure, the issue goes away.
Any help with this would be greatly appreciated. I'm happy to provide additional information if needed.
Debug Output:
I've passed -v -v -v -v to python above (python -v -v -v -v -m pytest . . .), and I see the following:
...
# trying /Users/<username>/git/proj/src/main/python/module_A/test_helpers.cpython-38-darwin.so
# trying /Users/<username>/git/proj/src/main/python/module_A/test_helpers.abi3.so
# trying /Users/<username>/git/proj/src/main/python/module_A/test_helpers.so
# trying /Users/<username>/git/proj/src/main/python/module_A/test_helpers.py
# trying /Users/<username>/git/proj/src/main/python/module_A/test_helpers.pyc
# trying /Users/<username>/git/proj/src/main/python/module_A/test_helpers.cpython-38-darwin.so
# trying /Users/<username>/git/proj/src/main/python/module_A/test_helpers.abi3.so
# trying /Users/<username>/git/proj/src/main/python/module_A/test_helpers.so
# trying /Users/<username>/git/proj/src/main/python/module_A/test_helpers.py
# trying /Users/<username>/git/proj/src/main/python/module_A/test_helpers.pyc
...
It appears as thought test_helpers is being looking for in ./src/main/python twice, but ./src/test/python isn't searched despite PYTHONPATH set to contain the ./src/test/python path in the command line above.
Example Test Code:
test_file.py
from pytest import mark
# This is where we have an error in the code editor despite
# this running fine.
#
# Import module_A.mocule_1.file could not be resolved
from module_A.test_helpers.helper_a import HelperA
#mark.unit_test
class Handler_tests:
def test_happy_path(self):
with HelperA():
pass
helper_a.py (located at ./src/test/python/module_A/test_helpers/)
class HelperA:
def __init__(self):
# No impl here
pass
Project Directory Structure:
// note, some non-python code directories/files exist but are omitted
<proj_root>
├──README.md
├──.env
└──src
├── main
│ └── python
│ └── module_A
│ ├── __init__.py
│ └── module_1
│ ├── __init__.py
│ └── file.py
└── test
└── python
├── conftest.py
└── module_A
├── __init__.py
└── test_helpers
├── __init__.py
└── helper_a
└── module_1
└── test_file.py
Environment Information:
Python
python ~3.8
pytest 6.0.0
VS Code
Version: 1.63.0-insider (Universal)
Commit: bedf867b5b02c1c800fbaf4d6ce09cefbafa1592
Date: 2021-11-18T05:17:00.890Z (3 wks ago)
Electron: 13.5.2
Chromium: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Darwin arm64 21.1.0
Pylance
v2021.12.2-pre.1
ok, your note gives me some insight what could be going on.
If you use import like that:
from module_A.test_helpers.helper_a import HelperA
It suggest your PYTHONPATH (so sys.path) contains both, src/main/python and src/test/python.
In order to import module, python looks on all paths in sys.path in order and search for specific module. Unfortunately it isn't smart enough to check full path - the first place it finds module_A it assumes it's the one and tries to resolve rest path. Your src/main/python path is first and for obvious reason it cannot find test_helpers there. It doesn't search another paths from sys.path and not sure is there a way to force it. The only solution I know is to have different names of files/directories under things you have in PYTHONPATH or use only one pointed to your src
I'm working on a python 3.8.5 project which has this folder structure:
project/
├── __init__.py
|── foo.py
|── bar.py
├── utils/
│ ├── __init__py
│ └── configurator.py
└── server/
├── __init__.py
├── models.py
In server/models.py I need to use a class Configurator declared inside utils/configurator.py:
# server/models.py
from utils.configurator import Configurator
# some code here ...
Unfortunately I get this error: ModuleNotFoundError: No module named 'utils'. I dont't understand how it cannot find the utils module since I've properly set the init.py file inside that folder.
I've also tried with relative import:
# server/models.py
from ..utils.configurator import Configurator
# some code here ...
this time getting the following error: ImportError: attempted relative import with no known parent package
Finally, I've attempted with an absolute import, with no results again:
# server/models.py
from projects.utils.configurator import Configurator
# some code here ...
The error this time is: ModuleNotFoundError: No module named 'projects'
Based on this previous question it seems correct to me, I can't understand what I'm messing with.
My code editor is Visual Studio Code and when I type from utils. ... it gives the right suggestion, although I don't think this piece of information could be of any utility.
The directory structure of my project looks like this:
project/
│
├── __init__.py
├── my_only_script.py
│
│
└── tests/
|
└── unit/
| ├── __init__.py
| └── unit_test.py
|
└── integration/
|
├── fixtures/
| ├── correct_data.csv
| └── generated_data.csv
|
├── __init__.py
└── test_integration.py
And I'm trying to execute my integration tests for my_only_script.py by running unittest in discover mode on the tests directory
# working directory is project root
python -m unittest discover -s tests
However, when I run this command, I'm met with the error
ImportError: Failed to import test module: integration.test_integration
Why is it that unittest could discover the integration test -- after all, it mentioned the module by name as integration.test_integration, so it must have found it -- but still, the integration test could not be imported?
edit
The full error message I encounter is
(base) [project]$ python -m unittest discover -s tests
E
======================================================================
ERROR: integration.test_integration (unittest.loader._FailedTest)
ImportError: Failed to import test module: integration.test_integration
Traceback (most recent call last):
File "/home/david/anaconda3/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
module = self._get_module_from_name(name)
File "/home/david/anaconda3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
import(name)
File "/home/david/project/tests/integration/test_integration.py", line 13
class PROJECT_INTEGRATION_TEST(unittest.TestCase)
^
SyntaxError: invalid syntax
Ran 1 test in 0.000s
FAILED (errors=1)
In test_integration.py, I have tried a relative import from ... import my_only_script, an absolute import from project import my_only_script, in addition to the plain-old import my_only_script. I've also tried commenting out the import statement entirely. All seem to result in the same error shown above.
I know there's nothing else wrong with the code in test_integration.py, because if I move it into the root of the project directory, and run it using python test_integration.py where test_integration.py uses import my_only_script the test runs without issue.
The problem was actually very simple; I left a colon off the end of
class PROJECT_INTEGRATION_TEST(unittest.TestCase) # <- no colon
in test_integration.py After fixing that, importing with import my_only_script in test_integration.py worked just fine.
My project structure is the following:
.
└── project name
├── project name
│ ├── __init__.py
│ ├── module.py
│
├── PACKAGE_A
│ ├── __init__.py
│ ├── PACKAGE_A.py
│ ├── module_a.py
│
In PACKAGE_A.py
from module_a import Some_Class
a = Some_Class()
class Another_Class:
# class code here
In module.py
"""
Notes
-----
https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
"""
# Standard library imports
import os, sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
# Local application imports
from PACKAGE_A.PACKAGE_A import Another_Class
from PACKAGE_A.module_a import some_function
While module_a.py and PACKAGE_A.py run without problem, running module.py fails with:
Traceback (most recent call last):
File "path\to\project name\project name\module.py", line 12, in <module>
from PACKAGE_A.PACKAGE_A import Another_Class
File "path\to\project name\PACKAGE_A\PACKAGE_A.py", line 1, in <module>
from module_a import Some_Class
ModuleNotFoundError: No module named 'module_a'
What am I doing wrong here?
You need to change your import statement in PACKAGE_A.py from:
from module_a import Some_Class
to:
from PACKAGE_A.module_a import Some_Class
The reason is that you are adding the path\to\project name\ to sys.path, but you have no module_a.py in path\to\project name\, and path\to\project name\PACKAGE_A (where module_a.py resides) is not in sys.path.
As for why you succeed in running everything in PACKAGE_A, it's because Python adds the directory containing the script you are running to the list (as explained by gaFF).
I would recommend you read a bit more about python imports, if the doc seems too cluttered, you can check this link.
This is a personal preference, but I find it simpler to add the root directory of the project to the PYTHONPATH environment variable and then running all the scripts from that directory's level and changing the import statements accordingly. In your example, the root directory would be path\to\project name\.
import search for your packages in specific places, listed in sys.path. See the doc for the full details.
The current directory is always appended to this list, that's why you succeed to run everything inside PACKAGE_A. But from project name, there is no way to know where to find PACKAGE_A.
Solutions include:
use relative import
always run from the root directory (and start all your imports from the root directory)
add the root directory to the environment variable PYTHONPATH (same)
use a tool which sets PYTHONPATH when you enter your virtual environment (same)
...
and depends on your project and your needs.
I realize that there are a lot of questions about this on StackOverflow already, but I find the solutions to be entirely unclear and often contradicting of each other.
I have the following scenario: I am writing a python (Python 3.4) package with the following structure:
mypackage/
__init__.py (empty file)
mydir1/
__init__.py (empty file)
mymodule1.py
mydir2/
__init__.py (empty file)
mymodule21.py
mymodule22.py
mydir3/
__init__.py (empty file)
mymodule31.py
mymodule32.py
tests/
__init__.py (empty file)
test_something_in_mydir1.py
test_something_in_mydir2.py
test_something_in_mydir3.py
I want the files within the tests/ directory to contain unit tests of everything inside the package. The problem is that no matter which approach I try, I get this error:
SystemError: Parent module '' not loaded, cannot perform relative import
I should note that I want this to work "out of the box" which means that I do not want to import things using their absolute paths and I do not want to change my path variable. None of these solutions seem very pythonic and its driving me slowly up the wall. Is there really no way to do this?
I've tried a bunch of stuff already, including examples below, but they all seem to produce the same result:
from ..mydir1.mymodule1 import *
from .mypackage.mydir1.mymodule1 import *
from ...mypackage.mydir1.mymodule1 import *
And I've tried overwriting the __all__ variable in each of the __init__.py files as well as doing the imports as shown here: https://docs.python.org/3/reference/import.html#submodules
Can anyone tell me how to structure the imports (for example) in the file test_something_in_mydir1.py in such a way as to have access to all the classes/functions that are found in mymodule1.py for unit testing the package?
If test_something_in_mydir1.py contains:
import unittest
from ..mydir1 import mymodule1
class Test1(unittest.TestCase):
def test1(self):
self.assertEqual(mymodule1.x,1) # mymodule1 trivially contains x=1
And if the unit tests were launched from the directory containing the package (in this case, C:\Test) using the following command:
py -m unittest discover -v
The tests are discovered and run correctly with relative imports:
C:\Test>tree /f
Folder PATH listing
Volume serial number is CE8B-D448
C:.
└───mypackage
│ __init__.py
│
├───mydir1
│ mymodule1.py
│ __init__.py
│
├───mydir2
│ mymodule21.py
│ mymodule22.py
│ __init__.py
│
├───mydir3
│ mymodule31.py
│ mymodule32.py
│ __init__.py
│
└───tests
test_something_in_mydir1.py
__init__.py
C:\Test>py -m unittest discover -v
test1 (mypackage.tests.test_something_in_mydir1.Test1) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK