Develop Reference

Trouble importing a module that imports a module

I'm having trouble with a python package that uses separate modules to structure code. The package itself is working, however when imported from another environment fails with a ModuleNotFound error.
Here's the structure:
Project-root
|
|--src/
| __init__.py
| module_a.py
| module_b.py
| module_c.py
| module_d.py
|--tests
etc.
In module_a.py I have:
from module_a import function_a1,...
from module_b import function_b1,...
from module_c import function_c1,...
In module_c I import module_d like:
from module_d import function_d1,...
As mentioned above, executing module_a or module_c directly from the CLI work as expected, the unit tests I've created in the test directory also work (with the help of sys.path.insert), however if I create a new environment and import the package I get the following error:
>>> import module_a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/<abs_path>/.venv/lib/python3.9/site-packages/module_a.py", line 22, in <module>
from module_c import function_c1, function_c2
File /<abs_path>/.venv/lib/python3.9/site-packages/module_c.py", line 9, in <module>
import module_d
ModuleNotFoundError: No module named 'module_d'
>>>
I've exhausted all ideas how to overcome this, besides combining the code of modules c and d in one file, which I'd hate to do, or rethink the flow so that all modules are imported from module_a.
Any suggestions how to approach this would be greatly appreciated.
Update: It turned out to be a typing mistake in the name of module_d in setup.py. For whatever reason python setup.py install was failing silently or I wasn't reading the logs carefully.

The problem comes down to understanding the basics of the import system and the PYTHONPATH.
When you try to import a module (import module_a), Python will search in order in every directory listed in sys.path. If a directory matches the name (module_a)1, then it runs the __init__.py file is such exist.
When you get an [https://docs.python.org/3/library/exceptions.html#ImportError], it means that there is no directory in sys.path containing a directory with the name asked.
You said for your tests you did something like sys.path.insert(0, "some/path/"), but it is not a solution, just a broken fix.
What you should do is set your PYTHONPATH environment variable to contain the directory where your modules are located, Project-root/src in your case. That way, no need to ever use sys.path.insert, or fiddle with relative/absolute paths in import statements.
When you create your new environment, just set your environment variable PYTHONPATH to include Project-root/src and you are done. This is how installing regular Python modules (libraries) work : they are all put into a directory in site-packages.
1: this changed since old Python versions, it used to be required for the directory to contain an __init__.py file

Python Module names are contradicting depending on where you run the code from

This is my folder structure -
From api.py I import IncidentHandler as below -
import data.IncidentHandler
data.IncidentHandler works fine HOWEVER - inside of the IncidentHandler.py I have the below present:
import BrowserHandler
When I run the code from api.py I get this error -
Traceback (most recent call last):
File "api.py", line 9, in <module>
import data.IncidentHandler
File "/User/**mask**/**mask**/**mask**/**mask**/tpptickethandler/src/data/IncidentHandler.py", line 1, in <module>
import BrowserHandler
ModuleNotFoundError: No module named 'BrowserHandler'
When I run the code directly from IncidentHandler.py it is working as expected.
I realised that the solution was in the modular name so I have changed as follows -
import BrowserHandler
to
import data.BrowserHandler
This now works from api.py but NOT from IncidentHandler.py
New Error from IncidentHandler.py
Traceback (most recent call last):
File "IncidentHandler.py", line 1, in <module>
import data.BrowserHandler
ModuleNotFoundError: No module named 'data'
I can see that I may be working against the typical Python principles however facing this issue was quite a unique challenge and I have not found anything on the internet thus far. Could someone please assist me with a solution and/or how to tackle this better next time.

You are trying to organize the modules under different folders, which are packages. For python to recognize a folder as a package, you need to have an __init__.py file under it, it doesn't matter if this file is empty or not, that will let python correctly recognize the folder as a package
https://docs.python.org/3/tutorial/modules.html#packages
example from the official site
sound/ Top-level package
__init__.py Initialize the sound package
formats/ Subpackage for file format conversions
__init__.py
wavread.py
wavwrite.py
aiffread.py
aiffwrite.py
auread.py
auwrite.py
...
effects/ Subpackage for sound effects
__init__.py
echo.py
surround.py
reverse.py
...
filters/ Subpackage for filters
__init__.py
equalizer.py
vocoder.py
karaoke.py
...

Having Trouble with Import Statement in Python

My current projects are structured like so.
python/
__init__.py
project_1/
__init__.py
program_1.py
project_2/
__init__.py
program_2.py
project_3/
__init__.py
program_3.py
I wanted to import a class from project_3/program_3.py, called INFO, from both project_1/program_1.py and project_2/program_2.py. In both my program_1.py and program_2.py, I've tried the following import statements that didn't work.
Edit - I typically "cd" into project_1 and run my program_1.py from there.
from project_3.program_3 import INFO
Traceback (most recent call last):
File "./program_1.py", line 43, in <module>
from project_3.program_3 import INFO
ImportError: No module named 'project_3'
from python.project_3.program_3 import INFO
Traceback (most recent call last):
File "./program_1.py", line 43, in <module>
from python.project_3.program_3 import INFO
ImportError: No module named 'python'
The only way for me to import class INFO into program_1.py and program_2.py is by copying program_3.py in both program_1 and program_2's folder. My import in program_1 and program_2's statement would become:
python/
__init__.py
project_1/
__init__.py
program_1.py
program_3.py
project_2/
__init__.py
program_2.py
program_3.py
project_3/
__init__.py
program_3.py
from program_3 import INFO
This is quite inconvenience because now I have to update program_3.py in all 3 locations. Assuming my the way I structured my project folder is retarded:
What's the correct/best way to structure my folders so that I could call program_3.py from program_1/2.py?
If there's nothing wrong with my current structure, how do I correctly import the INFO class into program_1/2.py?
I have read the python documentation but it didn't say anything regarding importing classes from one subfolder to another. I also looked at another post from Stack Overflow as well but it didn't have the same structure as mine.
Update 1 - It's important to note that I'm merely using project_1/2 as folder to organized my projects. I originally had init.py in all my folders because I have no clue what I'm doing. The only module/package that I wanted to import in my project_1/program_1.py is from project_3/program_3.py. Example:
python/
project_1/
program_1.py
project_2/
program_2.py
project_3/
__init__.py
program_3.py

When using submodules, you need to use relative imports. In the example provided, importing program_3 from program_1 should be :
from ..project_3 import program_3
and then you can use the INFO class: program_3.INFO.
This works if your executable file is outside the script, that is the file which uses the module python in the example should be outside the python module.
If parts of the python package are executed as a script, the -m option should be passed to the interpreter:
python3 -m python.project_1.program_1
I hope this helps.

The previous solution provides me the same error.
I found a fix by using sys:
import sys
sys.path.append('../')
from project_3 import program_3
...
This way, you setup 'python' folder as reference (in your case it's the parent folder but you can select the path you want)
I hope it helps,

Import modules from different folder (python)

I have a folder, which contains two separate folders, one of which holds some python modules, and the other one holds a python script that uses those modules:
parentFolder/
lib/
__init__.py
readFile.py
writeFile.py
folder/
run.py
The __init__.py file is empty. In run.py, I have the following:
from ..lib import readFile
data = readFile('file.dat')
This gives me the error
Traceback (most recent call last):
File "run.py", line 1, in <module>
from ..lib import readFile
ValueError: Attempted relative import in non-package
What am I missing?

You need to add __init__.py files (can be empty) to each of the directories to make them a package. See the documentation for more details.

Import object declared inside __init__.py

I'm having trouble understanding how objects declared inside '__init__.py' are/should be imported to other files.
I have a directory structure like so
top/
|
|_lib/
|_ __init__.py
|_ one.py
File contents are as follows
lib/__init__.py
a=object()
lib/one.py
from lib import a
Here is the problem. If I fire a python shell from top directory, the following command runs well
>>> from lib.one import a
However if I change directory to top/lib and fire a similar command in a new python shell, I get error.
>>> from one import a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "one.py", line 1, in <module>
from lib import a
ImportError: No module named lib
Ofcourse, I can change one.py like so, that will make everything work.
from __init__ import a
But I'm really trying to understand, why import command works from top directory and not from top/lib.
Thanks.

Generally speaking, I think it's best practice to have the data funnel up to __init__.py from the modules/subpackages rather than needing to rely on data from __init__.py in the surrounding modules. In other words, __init__.py can use one.py, but one.py shouldn't use data/functions in __init__.py.
Now, to your question...
It works in top because python does a relative import (which is gone in python3.x IIRC, so don't depend on it ;-). In other words, python looks in the current directory for a module or package name lib and it imports it. That's all fine so far. running from lib.one import a first imports lib (__init__.py) which works fine. Then it imports one -- lib still imports ok from one because it's relative to your current working directory -- Not relative to the source file.
When you move into the lib directory, python can no longer find lib in the current directory making it not importable. Note that with most packages, this is fixed by installing the package which puts it someplace that python can find it without it needing to be in the current directory.