Develop Reference

Why does error occurs when importing from same *sub*directory?

Consider this folder structure:
main.py
module_a/
aa.py
bb.py
__init__.py
In main.py, I import aa as:
from module_a import aa
aa.yyy()
Then in aa.py, I import bb and include its functions as:
import bb
bb.xxx()
However, when I run main.py, python says "No module named 'bb'".
May I know why this happens. What is the correct way to import bb.
Thanks!!!
I have tried to write aa.py as:
import .bb
bb.xxx()
But it still does not work.

why this happens
Because the aa folder is not a place that Python is searching for modules.
Python's imports are absolute by default. They only look in specific places determined by sys.path. In main.py, import module_a.aa works because the root folder of the project happens to be on the sys.path; that folder contains a module_a folder; and that folder contains aa.py.
What is the correct way to import bb.
Please use relative imports between files in your package. In this case, the necessary import in aa.py looks like:
from . import bb
Absolute imports are error-prone; a project that uses two packages whose contents have overlapping names will run into namespace collisions. (Sadly, the standard library uses absolute imports in most places, such that projects need to ban certain module names for safety.) Relative imports will also require much less maintenance, should you later rename a sub-package.
The only thing relative imports require is that the package gets loaded, which typically will happen automatically with the first (yes, absolute) import of any of the package contents. When the package is loaded, it automatically sets a __package__ attribute on the modules in that package, which Python can use to resolve the relative imports. It's important to note that relative imports are relative to the package hierarchy, not the directory structure, which is why this is necessary; imports like from .. import example do not work by figuring out the current file location and then going up a level in the directory hierarchy. Instead, they check the __package__ to figure out what the containing package is, then check the file/folder location for that, and work from there.
If the "driver" script is within the package, run it as a module, using the -m switch for Python. For example, from the root folder, if module_a/aa.py is the driver, use python -m module_a.aa. This instructs Python that module_a is the containing package for aa.py, and ensures it gets loaded even though no import in the code has loaded it.
Contrary to what many people will wrongly tell you, it is almost never required to manipulate sys.path; there are popular Python projects on GitHub, running hundreds of thousands of lines of code, which either do not use it at all or use it only once in an ancillary role (perhaps because of a special requirement for a documentation tool). Just don't do it.
Also contrary to what many people will wrongly tell you, __init__.py files are not required to create packages in Python. They are simply a place where additional code can be placed for package initialization - for example, to create aliases for sub-package contents, or to limit what will be imported with a *-import (by setting __all__).
import .bb
This is just invalid. Relative imports only use the from syntax. See above for the correct syntax.

Suppose the same file structure as stated in OP, and:
main.py:
import module_a.aa
module_a.aa.thisFile()
module_a.aa.module_a.bb.thisFile()
aa.py:
import module_a.bb
def thisFile():
print("aa")
bb.py:
def thisFile():
print("bb")
Then, this will print
aa
bb
like you would expect. The main difference here, is that bb.py is imported in aa.py via module_a.bb. Running main.py gives no problem, however, running aa.py does not work in this way. That is why you might want to add folders to your path, such that you can call a function from a different file without this trouble. This is done via:
import os, inspect, sys
current_folder = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parent_folder = os.path.dirname(current_folder)
sys.path.insert(0,parent_folder)
Then you can import your files such as import file. If you consider this option, I would suggest to do some reading about how this works. Tip: make sure you avoid cyclic import problems.

How to structure imports inside project to work for both scripts and modules?

I have a rather simple setup:
[FOLDER]
|-> [Lib]
__init__.py (__all__=["modA","modB"])
modA.py (contains class named classA)
modB.py (contains class named classB + from modA import classA)
test1.py (from Lib.modA import classA
from Lib.modB import classB)
|-> [example]
test2.py (import sys
sys.path.append("../")
from Lib.modA import classA
from Lib.modB import classB)
Running test1.py from the Lib folder works perfectly without errors. Running test2.py from the example folder on the other hand requires the sys-patch to find Lib at all; however, it then crashes with No module named modA tracing back to the from modA import classA in modB.py via from Lib.modB import classB in test2.py.
How is one supposed to define an import in a module such that it will also work irrespective of the possible location of any future script that may use/import said module?

Python programs should be thought of as packages and modules, not as directories and files. While there is some overlap, packages and modules are more restrictive but also better encapsulated as a result.
Mixing both – say by manually modifying sys.path – should only be done as a last resort.
TLDR:
Use qualified imports based on the package:
from Lib.modA import classA/from .modA import classA instead of from modA import classA .
Use the environment for discovery:
Add search paths via PYTHONPATH instead of sys.path.
Start by deciding which ones are the top-level packages.
This is the point where we go from "directories/files" to "package". Notably, we cannot go "above" the top-level later on, so it should contain everything we need. However, we cannot remove anything "below" either, so it should be a tight enough selection.
In the example we could go as low as treating modA and siblings as their own module-package, and as high as FOLDER encompassing the entire project.
[FOLDER]
|-> [Lib]
| |-> modA.py
: :
It is reasonable to pick Lib since it represents a self-contained part. Going higher to FOLDER would be excessive, going lower to modA and siblings would break their relation as belonging together.
Everything under the top-level package folder now belongs to the package.
Prominently, FOLDER/Lib/modA.py is now the module Lib.modA. It is not FOLDER.Lib.modA, nor just modA.
Import package content only with absolute or relative qualified names.
Now that the top-level is established, all import statements must be made with regard to it. Refer to modules by their qualified name only, even when two modules share a more common parent:
# Lib.modB
# okay, fully qualified import
from Lib.modA import classA
# broken, unqualified import
from modA import classA
In order to avoid typing out the entire fully qualified name, one can use a relative name instead. This reuses the current module name to derive the fully qualified name of the module to import.
# Lib.modB
# okay, relative import
from .modA import classA
Note: Relative imports are a package operation, not a filesystem operation. One cannot go beyond the top-level, but descend into package namespaces.
Everything inside the module is now encapsulated and self-contained.
All qualified imports will work irrespective of the location of the package itself. As long as the top-level can be imported, everything below it can be imported as well.
Notably, this works irrespective of the location of any future script that may use/import the package.
Run packaged code with absolute qualified names
When executing code inside a package, Python has to know that the code is part of the package. Thus, it must be run with its absolute qualified named. Use the -m switch to do so:
# import and execute Lib.test1
python3 -m Lib.test1
Enable packages via the environment, not the program.
The point of defining a package is getting a single, self-contained entity that represents a library. One could zip up a package or similar, and it would still be a package.
In return, this means that the code itself should not break this abstraction by directly adding/changing directories to search packages.
Instead of having scripts assume the location of the package, the environment - of the script, the user, or the entire machine – should expose the package. There are basically two ways to do this:
Announce the package location as a search location. This is suitable for development as it is flexible but hard to maintain. The PYTHONPATH is appropriate for this.
Move the package to a search location used by Python. This is suitable for production and distribution as it requires effort but is easy to maintain. Packaging and using a package manager is appropriate for this.

To what I understand, when you are trying to run manually there is no path set as the environment, and when you just do
python file.py
python doesn't know in which environment we are running, which is precisely why, we do sys.path.append, to add the sibling directory path to env.
In the case if you are using an IDE like spyder or pycharm. you have an option to set the directory in which your program is located, or in pycharm it creates the whole program as a package where all the paths are handled by IDE.

Unable to import class even though I already have __init__.py files

I'm trying to import a class in a different directory to another file, but can't seem to get it to work. I know this question has been asked a lot and I have looked through multiple stackoverflow solutions and at https://docs.python.org/3/tutorial/modules.html#packages
1: Importing files from different folder
2: import python file in another directory failed
I want to try to just use the method containing just __init__.py file instead of doing an import sys
My directory structure is as follows:
django_vue/
__init__.py
devices/
__init__.py
models.py
lib/
__init__.py
my_file.py
I'm trying to import the class Device from /django_vue/devices/models.py to /django_vue/lib/my_file.py by:
from devices.models import Device
However when I do that I still get the error:
from devices.models import Device
ModuleNotFoundError: No module named 'devices'
I'm not sure what I'm dong wrong since I already have the __init__ file in both directories. Any help is appreciated. Also I'm running python 3.6.

This is the folder structure I'm working with.
.
└── django_vue
├── devices
│   └── models.py
└── lib
└── file.py
When you run
$ python file.py
python has no way of knowing what's outside the directory.
python can't go back and then into devices/ just like that.
The easiest way to solve this would be to add the folder devices/ to sys.path. When python imports a module, it searches for the module from sys.path. Adding the path to devices/ would make it available for imports.
Here are my files.
# models.py
Device = 'device'
# file.py
import sys
sys.path.append('..') # adds the parent dir (which is django-vue/) to path
# django-vue dir has devices/ so now this is available for imports
# importing this works now
from devices.models import Device
print(Device)
Output
django_vue/lib$ python3 file.py
device

Think about it your are inside my_file.py and import something called devices.
How can python know where the name devices has come from.
It won't search your entire Drive for that module/package
Relative Import
use a relative import instead. write from ..devices.models import Device. This is telling python to go up one directory to the parent directory and that's where it will find the devices package. Your lib module should now work as a module
If you however run the my_file.py package directly (as in python C:/django_vue/lib/my_file.py)
You will still get an error. Not the same error; the new error will be something like
ImportError: attempted relative import with no known parent package
This is happening because you are actually running my_file.py
If you think about it why would you want to run my_file.py by itself when it is clearly a part of a package. Maybe you are just testing to see if the import works when you use your package. The problem with this is that it makes it seem like your packages relative imports don't work even though this actually works.
Create a main.py in django_vue and write from lib import my_file. This will run your my_file.py and you will notice there is no error.
What's happening here
Have you heard of __package__?
if you put print(str(__package__)) in your my_file.py and run my_file.py directly you will see that it prints None.
However if you run main.py (that you just created) you will see that when It reaches my_file.py, __package__ will actually be defined to something.
Ahhh... you see now it all makes sense; The error you originally got said something about no known parent package. If __package__ is undefined that means there is no relative parent package because the file was obviously run directly instead of as part of a package.
Consider Absolute imports
you also might want to consider using absolute imports because if you are working on the package you might change it directory structure while developing. so you need to keep changing the import references on the affected files.
Although you can find IDE's with python extensions that automatically to this as you change your directory. I believe VS Code does this automatically.

Replace the __init__ files with __main__.

Importing correctly with pytest

I just got set up to use pytest with Python 2.6. It has worked well so far with the exception of handling "import" statements: I can't seem to get pytest to respond to imports in the same way that my program does.
My directory structure is as follows:
src/
main.py
util.py
test/
test_util.py
geom/
vector.py
region.py
test/
test_vector.py
test_region.py
To run, I call python main.py from src/.
In main.py, I import both vector and region with
from geom.region import Region
from geom.vector import Vector
In vector.py, I import region with
from geom.region import Region
These all work fine when I run the code in a standard run. However, when I call "py.test" from src/, it consistently exits with import errors.
Some Problems and My Solution Attempts
My first problem was that, when running "test/test_foo.py", py.test could not "import foo.py" directly. I solved this by using the "imp" tool. In "test_util.py":
import imp
util = imp.load_source("util", "util.py")
This works great for many files. It also seems to imply that when pytest is running "path/test/test_foo.py" to test "path/foo.py", it is based in the directory "path".
However, this fails for "test_vector.py". Pytest can find and import the vector module, but it cannot locate any of vector's imports. The following imports (from "vector.py") both fail when using pytest:
from geom.region import *
from region import *
These both give errors of the form
ImportError: No module named [geom.region / region]
I don't know what to do next to solve this problem; my understanding of imports in Python is limited.
What is the proper way to handle imports when using pytest?
Edit: Extremely Hacky Solution
In vector.py, I changed the import statement from
from geom.region import Region
to simply
from region import Region
This makes the import relative to the directory of "vector.py".
Next, in "test/test_vector.py", I add the directory of "vector.py" to the path as follows:
import sys, os
sys.path.append(os.path.realpath(os.path.dirname(__file__)+"/.."))
This enables Python to find "../region.py" from "geom/test/test_vector.py".
This works, but it seems extremely problematic because I am adding a ton of new directories to the path. What I'm looking for is either
1) An import strategy that is compatible with pytest, or
2) An option in pytest that makes it compatible with my import strategy
So I am leaving this question open for answers of these kinds.

The issue here is that Pytest walks the filesystem to discover files that contain tests, but then needs to generate a module name that will cause import to load that file. (Remember, files are not modules.)
Pytest comes up with this test package name by finding the first directory at or above the level of the file that does not include an __init__.py file and declaring that the "basedir" for the module tree containing a module generated from this file. It then adds the basedir to sys.path and imports using the module name that will find that file relative to the basedir.
There are some implications of this of which you should beware:
The basepath may not match your intended basepath in which case the module will have a name that doesn't match what you would normally use. E.g., what you think of as geom.test.test_vector will actually be named just test_vector during the Pytest run because it found no __init__.py in src/geom/test/ and so added that directory to sys.path.
You may run into module naming collisions if two files in different directories have the same name. For example, lacking __init__.py files anywhere, adding geom/test/test_util.py will conflict with test/test_util.py because both are loaded as import test_util.py, with both test/ and geom/test/ in the path.
The system you're using here, without explicit __init__.py modules, is having Python create implicit namespace packages for your directories. (A package is a module with submodules.) Ideally we'd configure Pytest with a path from which it would also generate this, but it doesn't seem to know how to do that.
The easiest solution here is simply to add empty __init__.py files to all of the subdirectories under src/; this will cause Pytest to import everything using package/module names that start with directory names under src/.
The question How do I Pytest a project using PEP 420 namespace packages? discusses other solutions to this.

import looks in the following directories to find a module:
The home directory of the program. This is the directory of your root script. When you are running pytest your home directory is where it is installed (/usr/local/bin probably). No matter that you are running it from your src directory because the location of your pytest determines your home directory. That is the reason why it doesn't find the modules.
PYTHONPATH. This is an environment variable. You can set it from the command line of your operating system. In Linux/Unix systems you can do this by executing: 'export PYTHONPATH=/your/custom/path' If you wanted Python to find your modules from the test directory you should include the src path in this variable.
The standard libraries directory. This is the directory where all your libraries are installed.
There is a less common option using a pth file.
sys.path is the result of combining the home directory, PYTHONPATH and the standard libraries directory. What you are doing, modifying sys.path is correct. It is something I do regularly. You could try using PYTHONPATH if you don't like messing with sys.path

If you include an __init__.py file inside your tests directory, then when the program is looking to set a home directory it will walk 'upwards' until it finds one that does not contain an init file. In this case src/.
From here you can import by saying :
from geom.region import *
you must also make sure that you have an init file in any other subdirectories, such as the other nested test directory

I was wondering what to do about this problem too. After reading this post, and playing around a bit, I figured out an elegant solution. I created a file called "test_setup.py" and put the following code in it:
import sys, os
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
I put this file in the top-level directory (such as src). When pytest is run from the top-level directory, it will run all test files including this one since the file is prefixed with "test". There are no tests in the file, but it is still run since it begins with "test".
The code will append the current directory name of the test_setup.py file to the system path within the test environment. This will be done only once, so there are not a bunch of things added to the path.
Then, from within any test function, you can import modules relative to that top-level folder (such as import geom.region) and it knows where to find it since the src directory was added to the path.
If you want to run a single test file (such as test_util.py) instead of all the files, you would use:
pytest test_setup.py test\test_util.py
This runs both the test_setup and test_util code so that the test_setup code can still be used.

Are so late to answer that question but usining python 3.9 or 3.10 u just need to add __init__.py folder in tests folders.
When u add this file python interprets this folders as a module.
Wold be like this
src/
main.py
util.py
test/
__init__.py
test_util.py
geom/
vector.py
region.py
test/
__init__.py
test_vector.py
test_region.py
so u just run pytest.
Sorry my poor english

Not the best solution, but maybe the fastest one:
cd path/python_folder
python -m pytest python_file.py

How come my python import doesn't work?

This works:
from story.apps.document import core
print core.submit()
This doesn't work:
from story import apps
print apps.document.core.submit()
"story" is a directory. Inside it, there is "apps" directory. Inside it, there is "document" directory. "core.py" is a file.
There is a __init__.py in every directory.

The __init__.py file tells python to interpret the directory as a package, but it does not necessarily tell python to import sub-packages or other files from the directory (although it may, if you add the appropriate import statements).
With large package hierarchies, it is often preferable to require sub-packages to be imported explicitly.

When you do story.apps.document import core, you're telling the Python interpreter to match a module of the description story.apps.document, import it, then load the variable core from its namespace into your current one.
Because core is a file module it has in its namespace variables defined within that file e.g., submit.
When you do from story import apps, you're telling the Python interpreter to match a module of the description story, import it, then load the variable apps from its namespace into your current one.
Because apps is a directory module it has within its namespace variables defined in its __init__.py and other modules in that directory. So apps knows about the document but it doesn't know anything about document's submodule core.
FYI: The reason this sometimes confuses people is because of stuff like this...
Works just fine:
# File1
import story.apps.document
story.apps.document.core()
Doesn't work:
# File2
import story
story.apps.document.core() # <-- Looks like the same function call, but is an Error
For file1, The import works because the the import operation tries to intelligently find things on the filesystem. The function call works because the module document was imported, and it is merely named story.apps.document.
For file2, the function call doesn't work because there's nothing intelligent about the dot operator, it merely attempts to access attributes on the Python object--it doesn't know anything about filesystems or modules.

when you do from story import apps, you don't include all the subpackages inside apps, for that, you do something like,
from story.apps.document import *

This only works if story/__init__.py imports apps.

IN the second example you are only importing the package. Python does not automatically import subpackages or modules. You have to explicitly do it, as in the first example. Do a dir(apps) and you'll see it is just an empty package.