Develop Reference

Trouble importing a module that imports a module

I'm having trouble with a python package that uses separate modules to structure code. The package itself is working, however when imported from another environment fails with a ModuleNotFound error.
Here's the structure:
Project-root
|
|--src/
| __init__.py
| module_a.py
| module_b.py
| module_c.py
| module_d.py
|--tests
etc.
In module_a.py I have:
from module_a import function_a1,...
from module_b import function_b1,...
from module_c import function_c1,...
In module_c I import module_d like:
from module_d import function_d1,...
As mentioned above, executing module_a or module_c directly from the CLI work as expected, the unit tests I've created in the test directory also work (with the help of sys.path.insert), however if I create a new environment and import the package I get the following error:
>>> import module_a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/<abs_path>/.venv/lib/python3.9/site-packages/module_a.py", line 22, in <module>
from module_c import function_c1, function_c2
File /<abs_path>/.venv/lib/python3.9/site-packages/module_c.py", line 9, in <module>
import module_d
ModuleNotFoundError: No module named 'module_d'
>>>
I've exhausted all ideas how to overcome this, besides combining the code of modules c and d in one file, which I'd hate to do, or rethink the flow so that all modules are imported from module_a.
Any suggestions how to approach this would be greatly appreciated.
Update: It turned out to be a typing mistake in the name of module_d in setup.py. For whatever reason python setup.py install was failing silently or I wasn't reading the logs carefully.

The problem comes down to understanding the basics of the import system and the PYTHONPATH.
When you try to import a module (import module_a), Python will search in order in every directory listed in sys.path. If a directory matches the name (module_a)1, then it runs the __init__.py file is such exist.
When you get an [https://docs.python.org/3/library/exceptions.html#ImportError], it means that there is no directory in sys.path containing a directory with the name asked.
You said for your tests you did something like sys.path.insert(0, "some/path/"), but it is not a solution, just a broken fix.
What you should do is set your PYTHONPATH environment variable to contain the directory where your modules are located, Project-root/src in your case. That way, no need to ever use sys.path.insert, or fiddle with relative/absolute paths in import statements.
When you create your new environment, just set your environment variable PYTHONPATH to include Project-root/src and you are done. This is how installing regular Python modules (libraries) work : they are all put into a directory in site-packages.
1: this changed since old Python versions, it used to be required for the directory to contain an __init__.py file

Value Error: Attempted relative import beyond top-level package

I have been creating an application that will run just like Microsoft Notepad. My program comprises of the following file and folder:
libs (Folder)
scripts (Folder)
test.py
The test.py is the file in which I am trying the following code:
from .libs import tkinter
from .libs import pyglet
from .libs import threading
test_window = Tk()
test_window.mainloop()
In the folder libs I have added the modules that are required in the application.
Example: Tkinter, Pyglet and Threading
When I am trying to import them into test.py using the code above and then running the program. I see the following Traceback:
Traceback (most recent call last):
File "c:\Users\Bhavyadeep\Desktop\NuclearPad\test.py", line 1, in <module>
from .libs import tkinter
ImportError: attempted relative import with no known parent package
I don't really know how I solve this problem. Is it because I added the module in the folder where Import in not available? Or is it because I did something wrong in this module folder copying-pasting to a folder not at Python's PATH?

from .libs import tkinter
Means "import tkinter from the package called lib which is located in the same directory as file (this script)"
so your first line should say something like
test_window = tkinter.Tk()
unless you use either of the following:
from .libs.tkinter import *
from .libs.tkinter import Tk
The error is likely due to you using relative import syntax from within the script you are trying to execute. Remove the leading dots in order to execute test.py, otherwise append its directory to sys.path or use site.addsitedir(path_to_directory_containing_test_py) and import it as follows import test.
With that said, you should not call the module's file test.py because there is already something with that name in python's default Lib directory or builtins. If you insist, there should be a hacky way to get it done without changing the name through importlib.
As for why this error occurs, I think it's because the interpreter is designed to execute scripts but not modules/packages whose contents are placed in .pyc files in __pycache__s when imported.
In order to import from scripts or libs, they must be packages. In order to turn them into packages you must add __init__.py files to them.
Lastly, It's a pretty terrible idea to place third-party libraries in your source like this. At the very least you should have installed them properly on your machine and then copied their sources from site-packages/the equivalent for venvs, which will fail if they use any executables and/or have any external dependencies
Documentation for distributing python packages

Accessing homemade package from folder inside package, python

I've been working on two packages for a few months as an intern, and I'm wrapping it up to deliver it to the next intern to continue working on it.
I'm having problems with accessing the packages from inside a combined folder. My folder structure looks like this:
> final_packages
__init__.py
> stream_converter
__init__.py
matconverter.py
matloader.py
> stream_plotter
__init__.py
bits_params_manager.py
params_manager.py
plotter.py
folder_plots.py
> examples
example4.py
The __init__.py files in stream_converter and stream_plotter just imports all the modules present in each respective folder. The __init__.py file in Final_packages is empty.
I can use the packages from a file which is in the same folder as Final_packages when I include the directory of Final_packages to sys.path. However, if the file is included in the Examples folder, I get the following error:
>> import stream_converter
Traceback (most recent call last):
File "<ipython-input-23-2fa4258586e7>", line 1, in <module>
import stream_converter
File "C:\Users\kalse\OneDrive - Vestas Wind Systems A S\Personal Documents\Work_with_MORIK\Streaming_w_python\python3\Final_packages\stream_converter\__init__.py", line 1, in <module>
import matconverter
ModuleNotFoundError: No module named 'matconverter'
Somehow it seems that the python can't see the modules that are in the same folder as the __init__.py, even though it actually enters __init__.py. I would not expect it to be good practice t define a path inside these __init__.py files, so clearly I'm missing something - Can somebody help me here?
Best regards

Unable to import class even though I already have __init__.py files

I'm trying to import a class in a different directory to another file, but can't seem to get it to work. I know this question has been asked a lot and I have looked through multiple stackoverflow solutions and at https://docs.python.org/3/tutorial/modules.html#packages
1: Importing files from different folder
2: import python file in another directory failed
I want to try to just use the method containing just __init__.py file instead of doing an import sys
My directory structure is as follows:
django_vue/
__init__.py
devices/
__init__.py
models.py
lib/
__init__.py
my_file.py
I'm trying to import the class Device from /django_vue/devices/models.py to /django_vue/lib/my_file.py by:
from devices.models import Device
However when I do that I still get the error:
from devices.models import Device
ModuleNotFoundError: No module named 'devices'
I'm not sure what I'm dong wrong since I already have the __init__ file in both directories. Any help is appreciated. Also I'm running python 3.6.

This is the folder structure I'm working with.
.
└── django_vue
├── devices
│   └── models.py
└── lib
└── file.py
When you run
$ python file.py
python has no way of knowing what's outside the directory.
python can't go back and then into devices/ just like that.
The easiest way to solve this would be to add the folder devices/ to sys.path. When python imports a module, it searches for the module from sys.path. Adding the path to devices/ would make it available for imports.
Here are my files.
# models.py
Device = 'device'
# file.py
import sys
sys.path.append('..') # adds the parent dir (which is django-vue/) to path
# django-vue dir has devices/ so now this is available for imports
# importing this works now
from devices.models import Device
print(Device)
Output
django_vue/lib$ python3 file.py
device

Think about it your are inside my_file.py and import something called devices.
How can python know where the name devices has come from.
It won't search your entire Drive for that module/package
Relative Import
use a relative import instead. write from ..devices.models import Device. This is telling python to go up one directory to the parent directory and that's where it will find the devices package. Your lib module should now work as a module
If you however run the my_file.py package directly (as in python C:/django_vue/lib/my_file.py)
You will still get an error. Not the same error; the new error will be something like
ImportError: attempted relative import with no known parent package
This is happening because you are actually running my_file.py
If you think about it why would you want to run my_file.py by itself when it is clearly a part of a package. Maybe you are just testing to see if the import works when you use your package. The problem with this is that it makes it seem like your packages relative imports don't work even though this actually works.
Create a main.py in django_vue and write from lib import my_file. This will run your my_file.py and you will notice there is no error.
What's happening here
Have you heard of __package__?
if you put print(str(__package__)) in your my_file.py and run my_file.py directly you will see that it prints None.
However if you run main.py (that you just created) you will see that when It reaches my_file.py, __package__ will actually be defined to something.
Ahhh... you see now it all makes sense; The error you originally got said something about no known parent package. If __package__ is undefined that means there is no relative parent package because the file was obviously run directly instead of as part of a package.
Consider Absolute imports
you also might want to consider using absolute imports because if you are working on the package you might change it directory structure while developing. so you need to keep changing the import references on the affected files.
Although you can find IDE's with python extensions that automatically to this as you change your directory. I believe VS Code does this automatically.

Replace the __init__ files with __main__.

Importing a code-containing __init__.py from a cousin folder?

Given the directory structure:
program/
setup.py
ilm/
__init__.py
app/
__init__.py
bin/
script.py
Note: the setup.py is not a typical setup.py, rather it is a custom-made setup uniquely for py2app.
program/ilm/app/__init__.py is non-empty: it contains a main() function, which instantiates a class in the same file. My question: In program/ilm/bin/script.py, if I want to import and execute the main() function in program/ilm/app/__init__.py, what are the valid ways of achieving this? The reason I ask is that script.py is doing so thus:
import ilm.app as app
if __name__ == '__main__':
app.main()
Based on my (admittedly limited) understanding of packaging and importing, this shouldn't work, since we have not explicitly told script.py where to look for project/ilm/app/__init__.py using ... And indeed, I get:
MacBook-Pro-de-Pyderman:program Pyderman$ python ./bin/script.py
Traceback (most recent call last):
File "./bin/script.py", line 5, in <module>
import ilm.app as app
ImportError: No module named ilm.app
In contrast, when the Python interpreter is started in /project, import ilm.app as app works fine.
This is apparently fully-functional production code which I should not have to change to get running.
Is the import statement valid, given the directory structure, and if so, what am I missing?
If not, what is the recommended way of getting the import to work? Add the path using sys.path.append() above the import statement? Or use .. notation in the import statement to explicitly point to program to pick up project/ilm/app/__init__.py? Is the fact that it is an __init__.py I am trying to import significant?

Two things. You need to make sure the iml directory is in the python path. Either make sure you are running python from the right directory or add the right path to sys.path list. And you need to make sure that both iml and app directory both have
__init__.py
file, since python needs to interpret the whole thing as a hierarchy of modules rather than just dirs. Then you should be able to do
from iml import app

The obvious conclusion would seem to be that the iml directory has an __init__.py inside it, but why that would happen in your production setup is hard to say. Have you checked in the production environment whether this is the case?
Assuming that the production environment is importing the package at iml/app (which you can check by examining app.__file__) then the program will indeed execute the main function from the __init__.py file - but __init__.py might easily be importing it from sonewhere else rather than defining it locally.

The Python package needs proper setup.py which defines the package structure. Furthermore bin/ scripts should be defined as console_scripts entry points.
This all must be installed in a proper Python environment, not just any folder.