Develop Reference

How to properly dynamically import modules in packages for developing / testing

I have made an app with lots of packages and modules. I'm developing these packages (within their directories) and since I'm executing files within the package directory the import paths for modules/packages change.
For instance, the project looks a little bit like this...
├── app.py
└── utils
├── database.py
└── robot
├── __init__.py
├── run.py
├── recognition.py
└── housekeeping.py
If I executed python3 app.py then import utils.database is valid.
But if I was in the package and ran python3 run.py then import utils.database is not valid.
I want to know how to execute functions and import the modules/packages without getting an error.
I have got this working, but I don't think it is right. So far, I've tried using __name__ == "__main__" in every main .py file and using sys.path.append, but I feel like there must be an easier better-looking way.
This is what the inside of run.py looks like, I also had to do the same in database.py, is this the best way?
if __name__ == "__main__":
import sys
sys.path.append('../../')
from utils.robot.recognition import *
from utils.robot.housekeeping import *
import utils.database as db

Set the $PYTHONPATH environment variable to the directory containing utils, then your imports will work.
Never run a module contained within a package directly. It's problematic if the same is run directly and can also be imported from the package, because then there will be two copies of the module in the interpreter, which guarantees confusion.
Instead, use the -m interpreter option:
python -m utils.robot.run
Alternatively, if you never need to import run.py from other modules, you can remove the file from the package entirely and keep it in a separate 'scripts' directory, and then you don't need to use -m. But you still need to set up sys.path, either using $PYTHONPATH as before, or by doing sys.path manipulations. If you choose to do sys.path manipulations, then you should make it work regardless of what the current working directory is:
sys.path.append(os.path.join(os.path.dirname(__file__), '../../'))

Import a module from a sub package not working

My file structure is
project/
__init__.py
features/
__init__.py
CompareText.py
tests/
test.py
in test.py I am trying to import CompareText
from project.features import CompareText
I get an error of:
ModuleNotFoundError: No module named 'features'`
I checked the documentation and I think my import statement is correct. How can I fix it?

Add an __init__ file in test. Your project directory should look like this:
project/
__init__.py
features/
__init__.py
CompareText.py
tests/
__init__.py
test.py
Then in project/tests/test.py the following import statement will work:
from ..features import CompareText
Oh, and this will still raise an error if you try to run it directly. In the question you said you tried to import it like this:
from project.features import CompareText
This will only work if the parent directory of project is in Python's module search path. So, if you want to run the tests directly then modify the module search path as needed (See: sys.path).

Your import statement is supposed to look like this :
(But make sure your working directory is the same directory as your project folder is located during execution)
from project.features import CompareText
This is supposed to work if your current path while executing the script has the project folder
If you execute it while inside project folder you can use:
from .features import CompareText
Hope this helps!

I assume you are running test.py as a script. test.py needs to find the project package and two ways to do that are to make your project installable or to hack sys.path.
Installable
First, change your directory structure a bit so that project is a subdirectory of some anonymous directory you happen to be using for development. If you are checking this stuff into source control, it needs to be written so that it can be checked out anywhere. Move tests down one directory.
mydevdir/
setup.py
project/
__init__.py
features/
__init__.py
CompareText.py
tests/
test.py
How write a setup.py. This can get quite complicated. You can read Building and Distributing Packages with Setuptools and lookup other resources on the net, but a minimalist setup.py is
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(name='project',
version='0.1',
description='This is project: project',
packages=find_packages(),
)
Now, while in mydevdir do python setup.py develop. Or you can actually produce an install package and put it in a virtual env for test.
Hack sys.path
It may be easier to hack paths in test.py. Note that this will need to be undone if you make project installable later. Just add to the top of test.py
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).absolute().parents[2]))
This puts the parent directory in the python path and now project will be found on import. It runs the risk that a .py file in the same directory as project can mask an installed module. If you have a local csv.py and you import csv, you'd get the local.

Unit Testing Python; Imports breaking

I have a python app that I’m attempting to unit test. The code is sat in a src directory the tests in a test directory.
The problem comes when trying to test the class containing my main method. This class sits in the src directory and has references to classes in other modules that all sit in the src directory. In order to execute my app I have the references in this format from <module> import <class>
When I come to test this main class I write a test class and place it in the test directory. Running the test I find I have problems with the imports in the main class. To resolve I have to change the imports to be from from src.<module> import <class> Having done this my tests now pass but the app itself fails.
How can I resolve the issue so that my imports are valid for both my unit tests and normal execution?

It may fail due to the way import works.
I would recommend always using absolute imports. This means, that when creating a package, all imports within that package import all the way from the top level down: from package.module import thing. And more importantly, installing your package - not as a normal Python package, but with a symbolic link allows you to edit your source code while it still is considered an installed package. This way you can position your tests and scripts however you want them and the thing will always work.
Installing your package is done with pip install -e . from root. See a packaging tutorial for an example of packaging if you're not familiar with it.
An example package structure might be:
root/
setup.py
MANIFEST.in
...
package/
__init__.py
app.py
tests.py
module/
__init__.py
component.py
module_test.py
# Or have a tests folder under root
tests/
test_module.py
tests.py
And for example in app.py, and everywhere really:
from package.module.component import Class
from package.module import thing # that was declared in module/__init__.py
So to your case, if I understood it correctly, you'd have:
root/
setup.py
MANIFEST.in
...
src/
__init__.py
app.py
module.py
tests/
test_module.py
tests.py

trying to make paths work - attempted relative import beyond top-level package

I can't make this work..
My structure is:
program_name/
__init__.py
setup.py
src/
__init__.py
Process/
__init__.py
thefile.py
tests/
__init__.py
thetest.py
thetest.py:
from ..src.Process.thefile.py import sth
Running: pytest ./tests/thetest.py from program_name gives :
ValueError: attempted relative import beyond top-level package
I tried also other approaches but i am receiving various errors.
But I would expect for the above to work.

ValueError: Attempted relative import in non-package
States that you're trying to use relative import in the module, which are to be used for packages i.e. to make it a package add __init__.py and call the thetest.py from some file outside the package.
Directly running thetest.py from interpreter won't work.
Relative imports require that the module which uses them is being
imported itself either as package module.
Suggestion 1:
The current tests directory has a __init__.py file but that doesn't allow you to run it as a module (via shell) - to make your current (relative) import work, you need to import it in an external (to package) file/module - let's create a main.py (can name it anything you like):
main.py
program_name/
__init__.py
setup.py
src/
__init__.py
Process/
__init__.py
thefile.py
tests/
__init__.py
thetest.py
src/Process/thefile.py:
s = 'Hello world'
tests/thetest.py:
from ..src.Process.thefile import s
print s
main.py:
from program_name.tests.thetest import s
Executing main.py:
[nahmed#localhost ~]$ python main.py
Hello world
Suggestion 2:
Execute the file just above root dir i.e. one level up the program_name/ , in the following fashion:
[nahmed#localhost ~]$ python -m program_name.tests.thetest
Hell World
P.S. relative imports are for packages, not modules.

Just solved a similar problem with a lot of googling.
Here's two solutions without changing the existing file structor:
1
The way to import module from parent folder from ..src.Process.thefile.py import sth is called "relative import".
It's only supported when launching as a package from the top-level package. In your case, that is launching command line from the directory which contains program_name/ and type (for win environment)
python -m program_name.tests.thetest
or simply (useful for many pytest files):
python -m pytest
2
Otherwise -- when trying to run a script alone or from a non top-level package --
you could manually add directory to the PYTHONPATH at run time.
import sys
from os import path
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
from src.Process.thefile import s
Try the first one first see if it's compatiable with the pytest framework. Otherwise the second one should always solve the problem.
Reference (How to fix "Attempted relative import in non-package" even with __init__.py)

When importing a file, Python only searches the current directory, the directory that the entry-point script is running from.
you can use sys.path to include different locations
import sys
sys.path.insert(0, '/path/to/application/app/folder')
import thefile

Importing in __init__.py to a unittest package

I have a package and a test package. According to advice from Where do the Python unit tests go?, the tests should be in a different directory. The project's directory tree is as follows:
project\
kernel\
__init__.py
file1.py
file2.py
tests\
__init__.py
test1.py
test2.py
test3.py
I would like to import the kernel package to the tests package, because that is where file1.py and file2.py are being tested. Also, I would like to use one import statement in the __init__.py instead of importing kernel again and again in each test.
I tried adding the following to the __init__.py file in tests and to test2.py , test2.py (together and separately), with no success (the first does no harm, the second gives a syntax error):
import kernel
import ../kernel
I'm using python2.6. From command line all the above happens. When I use Eclipse PyDev, everything magically works.

The relative imports you're using will only work if the "project" directory is a python package (i.e. it has an __init__.py file in it). Try that first and see if that works for you.
If the kernel directory is acting as the "package" that would be distributed then you could put the tests directory inside of that and do the relative imports that way. So it would look like this:
project/
kernel/
__init__.py
file1.py
file2.py
tests/
__init__.py
test1.py ...
And you would import the kernel modules from the tests directory either as:
from kernel import file1 # if it's installed in the python path/environment
Or:
from .. import file1
# 'import ..file1' might work, but I'm not sure that's syntactically correct