Develop Reference

How to properly import between src and test modules

I have a project that I run from
root_dir> python ./src/main_file.py. When I want to test it, I use root_dir> python -m pytest and it will search for any files of the format test*.py or *test.py.
I have a file structure similar to the following:
root_dir
├── src
│ ├── __init__.py
│ ├── main_file.py
│ ├── file1.py
│ ├── file2.py
└── tests
├── __init__.py
└── file1_test.py
What I'm wanting to do is create a test file to test file1.py, but I'm getting a ModuleNotFoundError for what I believe is the following reason and I'm not fully sure what the proper way to address it is.
In file1_test.py:
./file1_test.py/
import src.file1 as file1
def test_file1():
<do stuff>
The problem occurs inside file1.py, where:
./file1.py/
import file2.py
<blah blah blah>
pytest fails with ModuleNotFoundError: No module named 'file2'. If I change file1.py to:
./file1.py/ (EDITED)
import src.file2.py
<blah blah blah>
then pytest will run successfully. However, if my main_file.py imports file1.py from within the same module (src),
./main_file.py/
import file1
<blah blah blah>
Then python ./src/main_file.py will fail with ModuleNotFoundError: No module named 'src.file2' due to that change. Changing file1 back to the original will work for main_file.py, but not pytest, and I'm stuck in a loop.
I can't figure out what I should be doing differently to get the normal import structure of main_file.py and pytest to cooperate. Should I be changing the directory? Should main_file.py be outside of src? I think I would just run into a similar issue where if file1 is importing file2, the import path a file outside of src expects and one inside of src will expect are always going to conflict.

There are three ways to do what you want.
Prefered way
root_dir
├── my_module
│ ├── __init__.py
│ ├── main_file.py
│ ├── file1.py
│ ├── file2.py
└── tests
├── __init__.py
└── file1_test.py
All files in my_module shoud import like
from my_module.file1 import some_function
Then you don't need to do any hacks
Keepin src and assuming statements like
from src.file1 import some_function
are not what you want you need to modify your tests like
import os
import sys
sys.path.insert(0, os.path.abspath('src')
Not modifying src, not modifying tests: Run tests with
PYTHONPATH=path/to/src pytest

How to add another python project to my project as a sub-model?

I copied python project B to my project A as a model. But B's include path is just based on B. So how can I call B's function from A. For example,
.
└── mypackage A
├── subpackage_1
│ ├── test11.py
│ └── test12.py
├── subpackage_2
│ ├── test21.py
│ └── test22.py
└── subpackage B
├── test31.py
└── test32.py
test31.py may include test32.py by
import test32
But from A's prospective, I should include it by
import B.test32
In fact, B is more complex than this example. How can I refactor it?

first make module using __init__.py in B
then try this:
from B.test32 import 'function Name'
and then call it.

As #Wàlid Bachri said, you should start placing __init__.py files, but it is a bit more complex than than.
You must put those __init__.py files, in each directory. So it would Look something like this.
.
└── mypackage A
├── __init__.py # here under A
├── subpackage_1
| ├── __init__.py
│ ├── test11.py
│ └── test12.py
├── subpackage_2
| ├── __init__.py # under package 2
│ ├── test21.py
│ └── test22.py
└── subpackage B
├── __init__.py # under package B
├── test31.py
└── test32.py
In each of the __init__.py files, you need to import all the files in the same directory. So for example in subpackage B you need to do this.
#/subpackage B/__init__.py
from . import test31
from . import test32
in mypackage A, where there are no files, just directories, you also do the same thing (from . import subpackage_2 etc).
If I suppose that mypackage A is the "main" package (as can be seen by your diargam), so it is not a submodule, to run any file you will need to execute the following.
First cd to the parent directory of mypackage A and then
# Suppose you want to execute /subpackage_1/test11
python -m mypackage_A.subpackage_1.test11 # WARNING mypackage A should have no whitespace
You may get a RuntimeWarning about your sys.modules being modified, but I can assure you, from experience, that you can safely ignore it. This is how python modules for pip are normally done, to ensure that pushing to production is easy.
EDIT : Also note that in all of your files, you should switch to package imports for relative imports, so you must use the dot syntax.
from . import test32 # this will import .test32
if you were to instead to import test32 from test31, the interpeter would try to search for a global package named test32 and not look in the same directory.

Simply add an empty python file called __init__.py to your B project to make Python treat this project as a normal python module.

Python no module named X - absolute imports

Before I even start I want to say - I know, that there are like bazillion questions similar to this, but I couldn't find the answer to my problem. I have a dir structure like this:
.
├── project
│   ├── A
│   │   ├── __init__.py
│   │   └── somelib.py
│   ├── B
│   ├── C
│   │   └── C
│   │   ├── foo.py
│   │   └── __init__.py
│   └── __init__.py
└── run.sh
run.sh:
python3 project/C/C/foo.py
foo.py:
from project.A.somelib import somefunc
VS Code actually gets the intellisense in foo.py - it tells me what funcitons/variables I can import from somelib. But when I run run.sh, I get this error message:
from project.A.somelib import somefunc
ModuleNotFoundError: No module named 'project'
Is there a way to solve this while preserving this directory structure?
adding project/__init__.py changed nothing
the sys.path in foo.py looks like this:
['/home/dabljues/projects/project/project/C/C', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/lib/python3.7/site-packages']
restrictions:
I can't modify neither sys.path in the files nor PYTHONPATH before running the scripts
I can't pip-install anything
I don't have sudo-access
I can't create a virtualenv, since the scripts are supposed to be downloadable and quickly executable

IDEs like VSCode or Pycharm make their own assumptions about a project, and will usually correctly link modules even if the interpreter that will ultimately run the code can't.
The reason why project.A.somelib can't be found is visible in your sys.path output, which gives you the places where python will search for modules. Since '/home/dabljues/projects/project/project' is not included, there is no way for python to resolve it during runtime.
A quick hack
You can just add the path manually to sys.path, either in the source file by running import sys; sys.insert(0, '/home/dabljues/projects/project/project/') in foo.py before any other imports happen, or by running export PYTHONPATH="${PYTHONPATH}:/home/dabljues/projects/project/project/" in your shell before run.sh.
Installing the project
Since it looks like you're developing a library, you might as well use the mechanisms python offers to make libraries shareable and thereby fixing any import issues. Add a minimal setup.py to the project root (i.e. /home/dabljues/projects/project/project/setup.py):
from setuptools import setup, find_packages
setup(
name='project',
version='0.1.0',
packages=find_packages('project'),
)
And install your project in editable mode:
$ python3 -m pip install -e .
This will put a link in your python3 executable's site-packages that points to the project root, which makes it accessible whenever you run anything with python3.
Tests
I included print(__name__) at the top of all python files to get some output.
running run.sh without installing the package:
$ sh run.sh
Traceback (most recent call last):
File "project/C/C/foo.py", line 1, in <module>
from project.A.somelib import somefunc
ModuleNotFoundError: No module named 'project'
after installing it
$ sh run.sh
__main__
project.A.somelib
As you can see, project.C.C.foo is executed as a script, yet it finds all imports that start with project because project is installed.

Run python in package mode helps.
1) Add __init__.py for every path:
.
├── project
│ ├── A
│ │ ├── __init__.py
│ │ └── somelib.py
│ ├── B
│ ├── C
│ │ ├── __init__.py
│ │ └── C
│ │ ├── foo.py
│ │ └── __init__.py
│ └── __init__.py
└── run.sh
2) Import module with relative path in foo.py:
from ...A.somelib import somefunc
3) Run python in package mode:
python -m project.C.C.foo
It works for me.

I'm unable to reproduce this given your code (assuming run.sh just starts a script).
Are you sure it's not a case of e.g. circular imports?
$ mkdir -p project/A project/C/C
$ cat > project/C/C/foo.py
print('moof!')
$ cat > project/A/somelib.py
print('beef!')
$ cat > script.py
import project.A.somelib
import project.C.C.foo
$ tree
.
├── project
│   ├── A
│   │   └── somelib.py
│   └── C
│   └── C
│   └── foo.py
└── script.py
$ python3 script.py
beef!
moof!

Change your run.sh script to run foo as a module. python3 -m proj.C.C.foo

Importing custom module into jupyter notebook

Yes, I know this is a recurrent question but I still couldn't find a convincing answer. I even read at https://chrisyeh96.github.io/2017/08/08/definitive-guide-python-imports.html but could not find out how to solve the problem:
I'm running python 3.6 project that includes jupyter (ipython) notebooks. I want the notebook to import a custom local helpers.py package that I will probably use also later in other sources.
The project structure is similar to:
my_project/
│
├── my_project/
│ ├── notebooks/
│ └── a_notebook.ipynb
│ ├── __init__.py # suppose to make package `my_project` importable
│ └── helpers.py
│
├── tests/
│ └── helpers_tests.py
│
├── .gitignore
├── LICENSE
├── README.md
├── requirements.txt
└── setup.py
When importing helpers in the notebook I get the error:
----> 4 import helpers
ModuleNotFoundError: No module named 'helpers'
I also tried from my_project import helpers and I get the same error ModuleNotFoundError: No module named 'my_project'
I finally (and temporarily) used the usual trick:
import sys
sys.path.append('..')
import helpers
But it looks awful and I'm still looking for a better solution

One can tell python where to look for modules via sys.path. I have a project structure like this:
project/
│
├── src/
│ └── my_module/
│ ├── __init__.py
│ └── helpers.py
├── notebooks/
│ └── a_notebook.ipynb
...
I was able to load the module like so:
import sys
sys.path.append('../src/')
from my_module import helpers
One should be able load the module from wherever they have it.

Here I could find several solutions. Some of them are similar to the ones answered before:
https://mg.readthedocs.io/importing-local-python-modules-from-jupyter-notebooks/index.html

If you move the notebooks directory out one level, and then explicitly import your module from the package, that should do it. So your directory would look like this:
my_project/
│
├── my_project/
│ ├── __init__.py
│ └── helpers.py
├── notebooks/
│ └── a_notebook.ipynb
...
and then your import statement within the notebook would be:
from my_project import helpers.

Try the following line:
from my_project.helpers import what_you_need
This line should also work:
import my_project.helpers

I think you need a __init__.py module in the notebooks/ directory. I haven't really used Jupyter notebooks before so I could be wrong. You may also need to try changing your import statement to:
import .. helpers
to indicate that the import statement is for a local package that is located in the parent directory of the Jupyter notebook.

This worked for me.
import sys
MODULE_FULL_PATH = '/<home>/<username>/my_project/my_project'
sys.path.insert(1, MODULE_FULL_PATH)
from my_module import helpers

If you are on a Unix/Linux system another elegant solution may be creating a "soft link" to the module file helpers.py that you would like to use. Change to the notebooks directory and create the link to the module file this way:
cd notebooks; ln -fs ../my_project/helpers.py .
This "soft link" is essentially a pointer (a shortcut) to the original target file. With the link in place you will be import your module file as usual:
import helpers

How to import python file from git submodule

I've a project which uses git submodules. In my python file I want to use functions from another python file in the submodule project.
In order to work I had to add the init.py file to all subfolders in the path. My folder tree is the following:
myproj
├── gitmodules
│   ├── __init__.py
│   ├── __init__.pyc
│   └── mygitsubmodule
│   ├── __init__.py
│   ├── __init__.pyc
│   └── file.py
└── myfile.py
Is there any way to make it work without touching mygitsubmodule ?
Thanks

you can add to sys.path in the file you want to be able to access the module, something like:
import sys
sys.path.append("/home/me/myproj/gitmodules")
import mygitsubmodule
This example is adding a path as a raw string to make it clear what's happening. You should really use the more sophisticated, system independent methods described below to determine and assemble the path.
Also, I have found it better, when I used this method, to use sys.path.insert(1, .. as some functionality seems to rely of sys.path[0] being the starting directory of the program.

I am used to avoiding modifying sys.path.
The problem is, when using git submodule, submodule is a project directory, not a Python package. There is a "gap" between your module and that package, so you can't import.
Suppose you have created a submodule named foo_project, and there is a foo package inside.
.
├── foo_project
│ ├── README.rst
│ └── foo
│ └── __init__.py
└── main.py
My solution will be creating a soft link to expose that package to your module:
ln -s foo_project/foo foo
.
├── foo_project
│ ├── README.rst
│ └── foo
│ └── __init__.py
├── foo -> foo_project/foo
└── main.py
Now you can import foo in the main.py.

For reference,
from submodulefolder.file import func_name
or
import submodulefolder.file as lib_name
where file excludes the extension of file.py, seems to work in relative terms without modifying the subfolder / git submodule with a init.py since python 3.3+,
as shown here.
Tested on py3.8.5 linux native and py3.7.8 anaconda Windows, both in Spyder's Ipython-console, as well as natively on linux via terminal.