I'm working on a project with this folder structure using pytest for tests
project
├── example
│ ├── __init__.py
│ └── example.py
├── setup.py
└── tests
└── test_1.py
I'm trying to import example.py from test_1.py but every way I've tried hasn't worked.
I've tried using sys.path.append('../) then import example and just using from ... import example but neither of these methods has worked. sys.path.append('../') didn't let me import example and from ... import example errored with ImportError: attempted relative import with no known parent package. I've also tried using python setup.py develop and it let me import example, but none of the methods are imported.
This is the code that is in test_1.py. It currently imports example but it has no content.
import sys
import os
sys.path.append("../example/")
import example
example.test()
Contents of example.py
class test:
def __init__(self):
self.a = 1
self.b = 2
The setup.py is the template from here
I think your problem is that the path resolution isn't functioning as you expect. In your example.py file do this:
import sys
import os
dir_path = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.realpath(f"{dir_path}/../example"))
import example
example.test()
Then, when in your root directory run python tests/test_1.py. This should execute test() from the example directory.
The realpath() function is the key here, as it will resolve the actual system path. If that isn't resolved it assumes whatever you're handing it is from the root (i.e. /example, which is wrong).
Use of sys.path.append results in flake8 reporting E402 module level import not at the top of the file. There are various hacks to this around which essentially depend on disabling E402; directly or by deception. There are at least two other ways.
Add __init__.py to the tests directory so that pytest loads the tests from within the parent dir and natively finds the include directory.
pip install -e . from within the parent and have the package loaded in a developer environment.
For the sake of consistency I started with uninstalling every package (so pip freeze returns nothing) and creating virtual environments for each of the approaches. I omitted the setup.py file favoring a minimal pyproject.toml file. Then pip install pytest and pip install flake8 in both. For packages, the second approach then shows an additional example package, i.e.
(.venv) PS C:\xxxx\Stackoverflow\proj_hatch> pip freeze
attrs==22.1.0
colorama==0.4.5
# Editable install with no version control (example==0.0.1)
-e c:\xxxx\stackoverflow\proj_hatch
flake8==5.0.4
iniconfig==1.1.1
mccabe==0.7.0
packaging==21.3
pluggy==1.0.0
py==1.11.0
pycodestyle==2.9.1
pyflakes==2.5.0
pyparsing==3.0.9
pytest==7.1.3
tomli==2.0.1
The net outcome is that both approaches have a working pytest.
(.venv) PS C:\xxxx\Stackoverflow\proj_hatch> pytest
================ test session starts =========================
platform win32 -- Python 3.10.8, pytest-7.1.3, pluggy-1.0.0
rootdir: C:\xxxx\Stackoverflow\proj_hatch
collected 1 item
tests\test_1.py . [100%]
=================== 1 passed in 0.01s ========================
(.venv) PS C:\xxxx\Stackoverflow\proj_hatch>
In both versions set example/__init__.py as below. This has an additional benefit of allowing the structure (which function in which file) to change later while keeping the import for other applications stable.
from example.example import test
__all__ = [
"test",
]
In both versions the test is:
import example
def test_1():
ex1 = example.Test()
assert ex1.b == 2
With the first approach it is necessary to add .\tests\__init__.py so that pytest loads the test from within the project directory and is able to find the example directory. I expect this has some implications for how you write tests.
With the second approach I started off using hatch however it seemed to work with little more than moving the example directory to src\example and by creating a pyproject.toml file. I've a suspicion there is some magic in the later versions of python that help. I am using 3.10.8 for this exercise.
[project]
name = "example"
version = "0.0.1"
Example:
sys.path.insert(0, "path_to_your_project")
Related
For reference, this is the opposite of this question. That question asks how to get pytest to use your local module. I want to avoid pytest using the local module.
I need to test my module's installation procedure. We should all be testing our modules' installation procedures. Therefore, I want my test suite to pretend like it's any other python module trying to import my module, which I have installed (whether or not it's up to date with my latest edits and/or an editable install is my business).
My project layout looks like this:
.
├── my_package
│ ├── __init__.py
│ └── my_module.py
├── setup.py
├── pyproject.toml
└── tests
├── conftest.py
├── __init__.py
└── test_my_package.py
If I
pip install .
cd tests
python -c "import my_package"
it all works. However, if I
pip install .
pytest
it does not. This is because pytest automatically adds the calling directory to the PYTHONPATH, making it impossible to test that pip install has worked. I want it to not do that.
I need to do this because I am using setuptools-scm (which has different behaviour in editable and non-editable installs) and setuptools.find_packages, which makes it easy to ignore subpackages. However, to reiterate, my issue is with pytest's discovery, not with the use of these two utilities.
See docs for pytest import mechanisms and sys.path/PYTHONPATH, which describe three import modes, which control this behavior. For instance, try this:
$ pytest --import-mode=importlib
which uses importlib to import test modules, rather than manipulating sys.path or PYTHONPATH.
A workaround is to manually edit the PYTHONPATH by changing tests/conftest.py to include
import sys
sys.path.pop(0)
before the first time my_module is imported, but it's not pretty and makes the assumption about where in the PYTHONPATH that item is going to show up. Of course, more code could be added to check explicitly, but that's really ugly:
import sys
from pathlib import Path
project_dir = str(Path(__file__).resolve().parent.parent)
sys.path = [p for p in sys.path if not p.startswith(project_dir)]
I need to write a package into a repository, but it is a small quick package, so I don't see the need to put files into a subdirectory. I simply want:
import mypkg.module1
with directory structure
root_folder/
- setup.py
- __init__.py # the init for package "mypkg" (empty)
- module1.py
And I don't want to be constrained to increase the folder hierarchy like here:
root_folder/
- setup.py
- mypkg/ # WHY would I do that for 1 module??
- __init__.py
- module1.py
# Content of setup.py
from setuptools import setup
setup(name='MyPkg',
packages=['mypkg']
package_dir={'mypkg': '.'})
but as a result, import mypkg fails, and import module1 works, instead of the desired import mypkg.module1.
I found this question, but the answer of "just move setup.py one folder up" does not suit me.
After finding setup argument documentation here, I tried to define a namespace package:
# Content of setup.py with a namespace package
from setuptools import setup
setup(name='MyPkg',
packages=['thisdir']
package_dir={'thisdir': '.'},
namespace_packages=['mypkg'])
Without testing, I think maybe something like that could work:
#!/usr/bin/env python3
import setuptools
setuptools.setup(
name='MyPkg',
version='0.0.0.dev0',
packages=['mypkg'],
package_dir={
'mypkg': '.',
},
)
The biggest issue is that, as far as I remember, you will end up with setup.py in the package mypkg as well, so that import mypkg.setup will indeed import the setup module. I have not verified.
The other big issue is that, as with most of the package_dir modifications, the "develop/editable" installation will most likely not work at all.
but as a result, import mypkg fails, and import module1 works, instead of the desired import mypkg.module1
I could imagine this happening, if the project is not actually installed, and the current working directory is the project's root directory.
For import mypkg to succeed you have to make sure the project is correctly installed (python -m pip install .).
And for import module1 to fail, you might need to change to a different directory.
Related:
Create editable package setup.py in the same root folder as __init__.py
Setting package_dir to ..?
I have been reading this, this and this but i still can't figure out how to proper import so i can run coverage.
This is the structure of my app
package/
app/
services/
__init__.py
news_services.py
test/
news_test.py
This is the import
from ..app.services.news_services import NewsServices
This is the command that i run the file.
coverage run .\test\news_test.py
And this is the error that i get
ValueError: attempted relative import beyond top-level package
EDIT: here is an answer for an app/service
For testing within the same app/service I often use this pattern:
import inspect
import os
import sys
test_dir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
src_dir = os.path.join(test_dir, '..', 'app')
sys.path.append(src_dir)
from services.news_services import NewsServices
EDIT: Below is the original for packages but the question appears to relate to an app/service
Your best bet, and the one I use on a daily basis with packages that I maintain, is to install your own package and not use relative imports for this at all.
Your tests will find and import the package exactly as your users would.
Here is my install.sh which you only have to do once; the trick is installing it in an "editable" fashion which, in this context, will cause anything importing it from the environment to find it exactly in your working structure with all your local changes:
#!/bin/bash
python3 -m venv env/
source env/bin/activate
pip install --editable "."
Then, within your tests:
from app.services.news_services import NewsServices
This is also described in this answer:
https://stackoverflow.com/a/6466139/351084
I'm trying to use pytest to check a function—here is my code:
# src/return_self.py
def return_self(n):
return n
# tests/return_self_test.py
import pytest
def test_1():
value = return_self(1)
assert value == 1
How do I require in my src file so that I can test it with pytest? I have tried a few things:
1. import return_self
2. from src.return_self import *
3. import sys
sys.path.append('../src')
4. import imp
return_self = imp.load_source('return_self', '/source/return_self.py')
I have also tried them with and without __init__.py files in the root and src directories. But each time, I get some variation on the error E ModuleNotFoundError: No module named 'return_self'. How can I require in my file?
You can try this approach:
# tests/return_self_test.py
import os
import sys
import pytest
sys.path.insert(1, os.path.join(sys.path[0], '..'))
from src.return_self import return_self
def test_1():
value = return_self(1)
assert value == 1
First, you must check that src/ and tests/ are in the same directory, I checked the function importing return_self in return_self_test.py and this is how:
Testpy search for files with test_[prefix] so I recommend changing return_self_test.py to test_return_self.py
# src/return_self.py
def return_self(n):
return n
# tests/test_return_self.py
import return_self
def test_1():
value = return_self.return_self(1)
assert value == 1
Finally, test in cmd (in the correct path) or Pycharm terminal with the following command:
py.test -v and voila! It's done (:
I will suggest you the setuptools approach (which makes your package distributable :D)
Project files' structure:
.
├── sample
│ ├── __init__.py
│ └── return_self.py
├── setup.cfg
├── setup.py
└── tests
└── test_return_self.py
where the sample/ directory matches the package's name and also must contain the source.
Minimal setup.py file content:
from setuptools import setup
setup(
setup_requires=['pytest-runner'],
tests_require=['pytest'],
name='sample'
)
Here we are configuring our test environment (you can extend the tests_require variable to include more testing requirements).
setup.cfg file content:
[aliases]
test=pytest
And here we specify we want to run the command pytest every time the developer does: python setup.py test
tests/test_return_self.py
from pytest import *
from sample.return_self import return_self
def test_return_self():
assert return_self(4) == 4
sample/return_self.py
def return_self(n):
return n
So, the next thing to do is to run:
python setup.py develop
to make your package available (while running the tests). If you're having problems with permission denied issues: append the --user option to the previous command to instruct python you wanna use the package without root permissions --in short the package will be installed into userland directories.
And finally run the tests using:
python setup.py test
Notes:
By using this approach, you'll be able to alter your project code on the fly (python setup.py develop needs to be run just once)
No ugly statements to inject the source directory into the current python path.
References:
Integrating with setuptools / python setup.py test / pytest-runner
Building and Distributing Packages with Setuptools
When using PyCharm to launch my tests I have no import problems at all, but if I do it in the terminal pytest complains about modules that cannot find. Precisely those under tests/tools.
Project folder structure:
src/
app/
tests/
end_to_end/ # End to end/Acceptance tests
tools/ # Helper modules used by some tests
unit # Unit tests
setup.py
Relevant parts of setup.py:
setup(...
packages=find_packages('src'),
package_dir={'': 'src'},
install_requires=['requests'],
test_suite='tests',
tests_require=[
'flexmock',
'pytest',
'wsgi_intercept'
])
Import line that fails when executing py.test tests/end_to_end, no matter if the virtualenv is active or not. Modules imported from the app package are fine:
from tests.tools.foomodule import Foo # Fails
from app.barmodule import Bar # Doesn't fail
I had to install my package with pip install -e packagename in order to be available for pytest when running tests from the command line. I suppose I will have to add the tools folder as a package in my setup.py, but this feels ugly to me because that modules are only for aiding in the testing process and aren't meant to be distributed.
On the other hand I could launch my tests with python setup.py test following the integration advices from pytest website.
probably you do not have __init__.py in your tests and/or tests/tools directory.
If missed python do see them as packages and is not able to import them.
You can:
add __init__.py (BAD see here
add tests/tools to your PYTHONPATH in your conftest.py and import a file (from import tests.tools.MODULE to import MODULE)
To be able to import modules residing outside src folder I went to "Project settings" in PyCharm, and there in "Project structure" I marked tests/end_to_end and tests/unit as source folders.
I also changed my mind and decided to put my helper modules inside those folders. This way I can import them directly and my tests run fine in terminal too because the current working directory is always added to the PYTHONPATH.