I have a bunch of unittest test cases in separate directories. There is also a directory which just contains helper scripts for the tests. So my file tree looks like this
test_dir1
test_dir2
test_dir3
helper_scripts
Each python file in test_dir* will have these lines:
import sys
sys.path.append('../helper_scripts')
import helper_script
This all works fine, as long as I run the tests from within their directory. However, I would like to be at the project root and just run:
py.test
and have it traverse all the directories and run each test it finds. The problem is that the tests are being run from the wrong directory, so the sys.path.append doesn't append the helper_scripts directory, it appends the parent of the project root. This makes all the imports fail with an Import Error.
Is there a way to tell py.test to run the test scripts from their directory? ie. change the cwd before executing them? If not, is there another test runner I can use that will?
What I usually do is structure my project like this:
myproject/
setup.py
myproject/
__init__.py
mymodule.py
tests/
__init__.py
test_dir1/
test_mymodule.py
helper_scripts/
__init__.py
helper_script.py
For running tests, I use a virtualenv with myproject installed in development mode using one of the following commands in the myproject root directory:
pip install -e .
python setup.py develop
Now in test_mymodule.py I can just say
from myproject.tests.helper_scripts import helper_script
I can then just run pytest and there's no need to change the working directory in tests at all.
See Pytest's Good Integration Practices for a great summary of pros and cons for different project directory structures.
os.chdir("newdir")
will change your current working directory
I would suggest that you instead configure your environment so that import helper_scripts will work regardless of the current directory. This is the recommended approach.
If you absolutely must though, you can use relative imports instead:
from .. import helper_script
Related
I have the following structure in my project;
project
src
├── A.py
└── B.py
tests
├── test_b.py
and in B.py I import A.py like this;
from A import foo
B.py works fine when I run it.
However when testing B.py in test_b.py I get an error saying
No module named A
I can make the test work with relative imports in B.py, but that fails when I run the module by itself.
Relative imports outside packages is a recipe for nightmares. Everything forks fine when you develop and test in the source directory. And problems start to occur as soon as you want to use your code from a different directory.
The workaround: Consistently add the directory of __file__ in sys.path before your local imports. As sys.path is a writable list, it will work. You should at least try to not add the directory if it is already present...
The idiomatic way: If you need local imports, then you probably need a package. It may require some work, because packages are expected to be installed, but it is a large + if you intend to later deploy your code. The downside, it that a package must be started as a module (python -m x.y) and not as a plain script (python x/y.py). With your current structure, I would just add an empty __init__.py file in both src and tests folder, and add a __main__.py file in src if you want to lauch directly the package.
Then you should run everything (including tests and dev runs) from project: python -m src.B [params...]. Same thing for the tests python -m tests.test_b. Or directly (as the test folder and files start with test): python -m unittest discover
First of all, there are a bunch of solutions on stackoverflow regarding this but from the ones I tried none of them is working. I am working on a remote machine (linux). I am prototyping within the dir-2/module_2.py file using an ipython interpreter. Also I am trying to avoid using absolute paths as the absolute path in this remote machine is long and ugly, and I want my code to run on other machines upon download.
My directory structure is as follows:
/project-dir/
-/dir-1/
-/__ init__.py
-/module_1.py
-/dir-2/
-/__ init__.py
-/module_2.py
-/module_3.py
Now I want to import module_1 from module_2. However the solution mentioned in this stackoverflow post: link of using
sys.path.append('../..')
import module_2
Does not work. I get the error: ModuleNotFoundError: No module named 'module_1'
Moreover, within the ipython interpreter things like import .module_3 within module_2 throws error:
import .module_3
^ SyntaxError: invalid syntax
Isn't the dot operator supposed to work within the same directory as well. Overall I am quite confused by the importing mechanism. Any help with the initial problem is greatly appreciated! Thanks a lot!
Why it didn't work?
If you run the module1.py file and you want to import module2 then you need something like
sys.path.append("../dir-2")
If you use sys.path.append("../..") then the folder you added to the path is the folder containing project-dirand there is notmodule2.py` file inside it.
The syntax import .module_3 is for relative imports. if you tried to execute module2.py and it contains import .module_3 it does not work because you are using module2.py as a script. To use relative imports you need to treat both module2.py and module_3.py as modules. That is, some other file imports module2 and module2 import something from module3 using this syntax.
Suggestion on how you can proceed
One possible solution that solves both problems is property organizing the project and (optionally, ut a good idea) packaging your library (that is, make your code "installable"). Then, once your library is installed (in the virtual environment you are working) you don't need hacky sys.path solutions. You will be able to import your library from any folder.
Furthermore, don't treat your modules as scripts (don't run your modules). Use a separate python file as your "executable" (or entry point) and import everything you need from there. With this, relative imports in your module*.py files will work correctly and you don't get confused.
A possible directory structure could be
/project-dir/
- apps/
- main.py
- yourlib/
-/__ init__.py
-/dir-1/
-/__ init__.py
-/module_1.py
-/dir-2/
-/__ init__.py
-/module_2.py
-/module_3.py
Notice that the the yourlib folder as well as subfolders contain an __init__.py file. With this structure, you only run main.py (the name does not need to be main.py).
Case 1: You don't want to package your library
If you don't want to package your library, then you can add sys.path.append("../") in main.py to add "the project-dir/ folder to the path. With that your yourlib library will be "importable" in main.py. You can do something like from yourlib import module_2 and it will work correctly (and module_2 can use relative imports). Alternatively, you can also directly put main.py in the project-dir/ folder and you don't need to change sys.path at all, since project-dir/ will be the "working directory" in that case.
Note that you can also have a tests folder inside project-dir and to run a test file you can do the same as you did to run main.py.
Case 2: You want to package your library
The previous solution already solves your problems, but going the extra mile adds some benefits, such as dependency management and no need to change sys.path no matter where you are. There are several options to package your library and I will show one option using poetry due to its simplicity.
After installing poetry, you can run the command below in a terminal to create a new project
poetry new mylib
This creates the following folder structure
mylib/
- README.rst
- mylib/
- __init__.py
- pyproject.toml
- tests
You can then add the apps folder if you want, as well as subfolders inside mylib/ (each with a __init__.py file).
The pyproject.toml file specifies the dependencies and project metadata. You can edit it by hand and/or use poetry to add new dependencies, such as
poetry add pandas
poetry add --dev mypy
to add pandas as a dependency and mypy as a development dependency, for instance. After that, you can run
poetry build
to create a virtual environment and install your library in it. You can activate the virtual environment with poetry shell and you will be able to import your library from anywhere. Note that you can change your library files without the need to run poetry build again.
At last, if you want to publish your library in PyPi for everyone to see you can use
poetry publish --username your_pypi_username --password _passowrd_
TL; DR
Use an organized project structure with a clear place for the scripts you execute. Particularly, it is better if the script you execute is outside the folder with your modules. Also, don't run a module as a script (otherwise you can't use relative imports).
When using PyCharm to launch my tests I have no import problems at all, but if I do it in the terminal pytest complains about modules that cannot find. Precisely those under tests/tools.
Project folder structure:
src/
app/
tests/
end_to_end/ # End to end/Acceptance tests
tools/ # Helper modules used by some tests
unit # Unit tests
setup.py
Relevant parts of setup.py:
setup(...
packages=find_packages('src'),
package_dir={'': 'src'},
install_requires=['requests'],
test_suite='tests',
tests_require=[
'flexmock',
'pytest',
'wsgi_intercept'
])
Import line that fails when executing py.test tests/end_to_end, no matter if the virtualenv is active or not. Modules imported from the app package are fine:
from tests.tools.foomodule import Foo # Fails
from app.barmodule import Bar # Doesn't fail
I had to install my package with pip install -e packagename in order to be available for pytest when running tests from the command line. I suppose I will have to add the tools folder as a package in my setup.py, but this feels ugly to me because that modules are only for aiding in the testing process and aren't meant to be distributed.
On the other hand I could launch my tests with python setup.py test following the integration advices from pytest website.
probably you do not have __init__.py in your tests and/or tests/tools directory.
If missed python do see them as packages and is not able to import them.
You can:
add __init__.py (BAD see here
add tests/tools to your PYTHONPATH in your conftest.py and import a file (from import tests.tools.MODULE to import MODULE)
To be able to import modules residing outside src folder I went to "Project settings" in PyCharm, and there in "Project structure" I marked tests/end_to_end and tests/unit as source folders.
I also changed my mind and decided to put my helper modules inside those folders. This way I can import them directly and my tests run fine in terminal too because the current working directory is always added to the PYTHONPATH.
I'm trying to use nosetests to run my tests in a directory structure like this
src
- file1.py
- ...
test
- helper.py
- test_file1.py
As you can see, test_file1.py has some functions that test file1.py, so it imports file1.py like this:
# In file1.py
import file1
import helper
# Tests go here...
I also use a helper.py file that has some neat functionality built in so that I can create tests more easily. This functionality is achieved by extending a couple of classes in my actual code and overriding some methods. So helper.py looks something like this:
# In helper.py
import file1
# Use stuff in file1.py
I'm having trouble understanding how nose goes about importing these things with its custom importer. I was able to get my test file to import file1.py by running nosetest ../tests within the src directory, but I'm currently getting an error akin to:
File helper.py:
ImportError: cannot import name file1
How does nose do its imports and is there a way I can essentially get it to lump all my tests/src files together so they can all import one another while I keep them in separate folders?
Seeing that you execute tests with nosetests ../tests I assume they are executed from the tests folder itself. Therefore, files from the src directory are not added to sys.path, hence the error.
To fix this one could:
run tests from the parent directory - nosetests will be able to identify src and test (or tests) directory by himself and will add them to the sys.path before running tests
add src directory path to the PYTHONPATH before running nosetests (export PYTHONPATH=../src; nosetests)
Note that you can as well omit the last argument to the nosetests as by default it runs the tests from current directory. Otherwise, if the tests are not in the directory you launch nosetests from, you can define its location with --where=<path-to-tests> parameter (or, simply -w). So for example you can execute tests from src direcotory and without even setting the PYTHONPATH (because current directory will be added to sys.path by default) like this: nosetests -w ../tests.
Lastly, even though this is very questionable by itself, and yet: the most common way to organize a Python source code is having python files and packages starting directly in the project directory, and having tests in "test" sub-packages of the packages they test. So, in your case it would be:
/file1.py
/test/helper.py
/test/test_file1.py
or better:
/myproject/__init__.py
/myproject/file1.py
/myproject/test/__init__.py
/myproject/test/helper.py
/myproject/test/test_file1.py
(latter, provided you also use correct imports in your test sources, e.g. from .. import file1).
In which case one runs tests from the project's root directory simply with nosetests without any argument.
Anyway, nosetests is flexible enough to work with any structure - use whatever seems more suitable for you and the project.
More on project structure in What is the best project structure for a Python application?
This seems generally like an issue I had with nose tests:
Importing with Python and Nose Tests
The work around I found was to insert a try..except block so that BOTH python and nosetest commands will work on the same directory as follows:
(1) In your main file, at the very top before anything else add:
# In file1.py
try:
# This will allow you to do python file1.py inside the src directory
from file2 import *
from helper import *
except:
# This will allow you to run nosetests in the directory just above
# the src and test directories.
from src.file1 import *
from src.helper import *
(2) Inside your test.py file add:
from src.file2 import *
from src.helper import *
I have some problems in structuring my python project. Currently it is a bunch of files in the same folder. I have tried to structure it like
proj/
__init__.py
foo.py
...
bar/
__init__.py
foobar.py
...
tests/
foo_test.py
foobar_test.py
...
The problem is that I'm not able, from inner directories, to import modules from the outer directories. This is particularly annoying with tests.
I have read PEP 328 about relative imports and PEP 366 about relative imports from the main module. But both these methods require the base package to be in my PYTHONPATH. Indeed I obtain the following error
ValueError: Attempted relative import in non-package.
So I added the following boilerplate code on top of the test files
import os, sys
sys.path.append(os.path.join(os.getcwd(), os.path.pardir))
Still I get the same error. What is the correct way to
structure a package, complete with tests, and
add the base directory to the path to allow imports?
EDIT As requested in the comment, I add an example import that fails (in the file foo_test.py)
import os, sys
sys.path.append(os.path.join(os.getcwd(), os.path.pardir))
from ..foo import Foo
When you use the -m switch to run code, the current directory is added to sys.path. So the easiest way to run your tests is from the parent directory of proj, using the command:
python -m proj.tests.foo_test
To make that work, you will need to include an __init__.py file in your tests directory so that the tests are correctly recognised as part of the package.
I like to import modules using the full proj.NAME package prefix whenever possible. This is the approach the Google Python styleguide recommends.
One option to allow you to keep your package structure, use full package paths, and still move forward with development would be to use a virtualenv and put your project in develop mode. Your project's setup.py will need to use setuptools instead of distutils, to get the develop command.
This will let you avoid the sys.path.append stuff above:
% virtualenv ~/virt
% . ~/virt/bin/activate
(virt)~% cd ~/myproject
(virt)~/myproject% python setup.py develop
(virt)~/myproject% python tests/foo_test.py
Where foo_test.py uses:
from proj.foo import Foo
Now when you run python from within your virtualenv your PYTHONPATH will point to all of the packages in your project. You can create a shorter shell alias to enter your virtualenv without having to type . ~/virt/bin/activate every time.