I have the following structure in my project;
project
src
├── A.py
└── B.py
tests
├── test_b.py
and in B.py I import A.py like this;
from A import foo
B.py works fine when I run it.
However when testing B.py in test_b.py I get an error saying
No module named A
I can make the test work with relative imports in B.py, but that fails when I run the module by itself.
Relative imports outside packages is a recipe for nightmares. Everything forks fine when you develop and test in the source directory. And problems start to occur as soon as you want to use your code from a different directory.
The workaround: Consistently add the directory of __file__ in sys.path before your local imports. As sys.path is a writable list, it will work. You should at least try to not add the directory if it is already present...
The idiomatic way: If you need local imports, then you probably need a package. It may require some work, because packages are expected to be installed, but it is a large + if you intend to later deploy your code. The downside, it that a package must be started as a module (python -m x.y) and not as a plain script (python x/y.py). With your current structure, I would just add an empty __init__.py file in both src and tests folder, and add a __main__.py file in src if you want to lauch directly the package.
Then you should run everything (including tests and dev runs) from project: python -m src.B [params...]. Same thing for the tests python -m tests.test_b. Or directly (as the test folder and files start with test): python -m unittest discover
Related
I have a big project:
main
golang
src
file1.go
python
src
file1.py
file2.py
__init__.py
java
src
file1.java
scripts
script.py
validator.sh
venv
bin
pip
python3
pyyaml
dateutil
Python project will use interpreter from:
ven/bin/python3
So anywhere inside
file1.py
file2.py
I can use imports:
import pyyaml
import dateutil
And this will work, by running from CLI:
venv/bin/python3 python/src/file1.py
However I wish to use some functions from file1.py inside file2.py
And have "relative reference" like this (inside file2.py)
from src.file1 import some_function
But having this kind of import and running the same way as before from CLI fails with error:
ModuleNotFoundError: No module named 'src'
What should I do? Pay attention that I have init.py file.
When you do from src.file1 this is relative to your sys.path.
Usually your current working directory is the first element in sys.path.
Thus you need to cd to main/python and then run ../venv/bin/python3 src/file1.py to make the imports work.
I've had similar issues and thus I have created an experimental, new import library: ultraimport.
It gives you more control over your imports and allows file system based imports.
You could then write in file2.py:
import ultraimport
file1 = ultraimport('__dir__/file1.py)'
This will always work, no matter how you run your code
The following is my directory structure.
ankur
├── ankur1
│ ├── __init__.py
│ └── util.py
├── ankur2
│ └── main.py
└── __init__.py
In main.py, I am importing the following.
import ankur.ankur1.util
When I execute the code in Windows, it works perfectly fine. But in Linux, I get the following error.
ImportError: No module named ankur.ankur1.util
I also read the official python doc on Modules and Packages.
Your package structure is OK. Your import statement is OK. The only thing missing is for the package to be visible in sys.path, a list of locations where import statements can be resolved.
Usually we do this by "installing" the package locally with pip, which copies your code into site-packages†. This directory is one of the entries in sys.path, so when your code is installed in site-packages, the import statements can now be resolved as usual.
However, to install your code you'll need an installer (setup.py script) or a build system (pyproject.toml file) defined for the package. Your project doesn't appear to have any installer or build system, so you'll need to create one (see the Python Packaging User Guide for details about that) and then install the package with pip. If you don't want to learn Python packaging just yet, you'll need to find another way around.
It is possible to modify sys.path directly in main.py, which is subsequently enabling the statement import ankur.ankur1.util to be resolved. This is hacky and I recommend against that. It would add the restriction that executing main.py is the only entry point to the rest of the package, and so any other code wanting to import ankur will first need to know the path to main.py on the filesystem. That's a messy approach and should be avoided.
Another way around is to use the environment - there is an environment variable PYTHONPATH which can be used to augment the default search path for module files. In your shell:
export PYTHONPATH=/path/to/parent # linux/macOS
SET PYTHONPATH=C:/path/to/parent # Windows
Where parent is the directory containing ankur subdirectory.
† The exact location of site-packages depends on your OS/platform, but you can check with import sysconfig; sysconfig.get_paths()["purelib"]
Given the directory structure:
/home/user/python/mypacakge/src/foo.py
/home/user/python/mypacakge/tests
/home/user/python/mypacakge/tests/fixtures
/home/user/python/mypacakge/tests/fixtures/config.json.sample
/home/user/python/mypacakge/tests/foo_tests.py
/home/user/python/mypacakge/README.md
Where src contains the source code, and test contains the unit tests, how do I setup a "package" so that my relative imports that are used in the unit tests located in test/ can load classes in src/?
Similar questions: Python Relative Imports and Packages and Python: relative imports without packages or modules, but the first doesn't really answer my question (or I don't understand it) and the second relies on symlinks to hack it together (respectively).
I figured it out.
You have to have __init__.py in each of the folders like so:
/home/user/python/mypackage/src/__init__.py
/home/user/python/mypackage/src/Foo.py
/home/user/python/mypackage/tests
/home/user/python/mypackage/tests/fixtures
/home/user/python/mypackage/tests/fixtures/config.json.sample
/home/user/python/mypackage/tests/foo_test.py
/home/user/python/mypackage/tests/__init__.py
/home/user/python/mypackage/README.md
/home/user/python/mypackage/__init__.py
This tells python that we have "a package" in each of the directories including the top level directory. So, at this point, I have the following packages:
mypackage
mypackage.test
mypackage.src
So, because python will only go "down into" directories, we have to execute the unit tests from the root of the top-most package, which in this case is:
/home/user/python/mypackage/
So, from here, I can execute python and tell it to execute the unittest module and then specify which tests I want it to perform by specifying the module using the command line options
python -m unittest tests.foo_test.TestFoo
This tells python:
Execute python and load the module unittest
Tell unit test to run the tests contained in the class TestFoo, which is in the file foo_test.py, which is in the test directory.
Python is able to find it because __init__.py in each of these directories promotes them to a package that python and unittest can work with.
Lastly, foo_test.py must contain an import statement like:
from src import Foo
Because we are executing from the top level directory, AND we have packages setup for each of the subdirectories, the src package is available in the namespace, and can be loaded by a test.
I have a bunch of unittest test cases in separate directories. There is also a directory which just contains helper scripts for the tests. So my file tree looks like this
test_dir1
test_dir2
test_dir3
helper_scripts
Each python file in test_dir* will have these lines:
import sys
sys.path.append('../helper_scripts')
import helper_script
This all works fine, as long as I run the tests from within their directory. However, I would like to be at the project root and just run:
py.test
and have it traverse all the directories and run each test it finds. The problem is that the tests are being run from the wrong directory, so the sys.path.append doesn't append the helper_scripts directory, it appends the parent of the project root. This makes all the imports fail with an Import Error.
Is there a way to tell py.test to run the test scripts from their directory? ie. change the cwd before executing them? If not, is there another test runner I can use that will?
What I usually do is structure my project like this:
myproject/
setup.py
myproject/
__init__.py
mymodule.py
tests/
__init__.py
test_dir1/
test_mymodule.py
helper_scripts/
__init__.py
helper_script.py
For running tests, I use a virtualenv with myproject installed in development mode using one of the following commands in the myproject root directory:
pip install -e .
python setup.py develop
Now in test_mymodule.py I can just say
from myproject.tests.helper_scripts import helper_script
I can then just run pytest and there's no need to change the working directory in tests at all.
See Pytest's Good Integration Practices for a great summary of pros and cons for different project directory structures.
os.chdir("newdir")
will change your current working directory
I would suggest that you instead configure your environment so that import helper_scripts will work regardless of the current directory. This is the recommended approach.
If you absolutely must though, you can use relative imports instead:
from .. import helper_script
I have a project directory that is set up in the following way:
>root
> modules
__init__.py
module1.py
> moduleClass
__init__.py
moduleClass1.py
moduleClass2.py
> scripts
runTests.py
> tests
__init__.py
test1.py
test2.py
run.sh
In runTests.py I have the following import statements:
import modules.module1
import modules.moduleClass.moduleClass2
import tests.test1
import tests.test2
The first two import statements work fine, but the second two give me the errors ImportError: No module named test1 and ImportError: No module named test2. I can't see what is different between the tests and modules directories.
I'd be happy to provide more information as needed.
When you run a script, Python adds the script's containing directory (here, scripts/) to sys.path. If your modules don't appear in sys.path any other way, that means Python may not be able to find them at all.
The usual solution is to put your scripts somewhere in your module hierarchy and "run" them with python -m path.to.module. But in this case, you could just use an existing test runner: Python comes with python -m unittest discover, or you might appreciate something fancier like py.test (pip install --user pytest).
The problem turned out to be that python didn't like the folder name tests. Changing the name to unit_tests solved the problem.