Develop Reference

Why does error occurs when importing from same *sub*directory?

Consider this folder structure:
main.py
module_a/
aa.py
bb.py
__init__.py
In main.py, I import aa as:
from module_a import aa
aa.yyy()
Then in aa.py, I import bb and include its functions as:
import bb
bb.xxx()
However, when I run main.py, python says "No module named 'bb'".
May I know why this happens. What is the correct way to import bb.
Thanks!!!
I have tried to write aa.py as:
import .bb
bb.xxx()
But it still does not work.

why this happens
Because the aa folder is not a place that Python is searching for modules.
Python's imports are absolute by default. They only look in specific places determined by sys.path. In main.py, import module_a.aa works because the root folder of the project happens to be on the sys.path; that folder contains a module_a folder; and that folder contains aa.py.
What is the correct way to import bb.
Please use relative imports between files in your package. In this case, the necessary import in aa.py looks like:
from . import bb
Absolute imports are error-prone; a project that uses two packages whose contents have overlapping names will run into namespace collisions. (Sadly, the standard library uses absolute imports in most places, such that projects need to ban certain module names for safety.) Relative imports will also require much less maintenance, should you later rename a sub-package.
The only thing relative imports require is that the package gets loaded, which typically will happen automatically with the first (yes, absolute) import of any of the package contents. When the package is loaded, it automatically sets a __package__ attribute on the modules in that package, which Python can use to resolve the relative imports. It's important to note that relative imports are relative to the package hierarchy, not the directory structure, which is why this is necessary; imports like from .. import example do not work by figuring out the current file location and then going up a level in the directory hierarchy. Instead, they check the __package__ to figure out what the containing package is, then check the file/folder location for that, and work from there.
If the "driver" script is within the package, run it as a module, using the -m switch for Python. For example, from the root folder, if module_a/aa.py is the driver, use python -m module_a.aa. This instructs Python that module_a is the containing package for aa.py, and ensures it gets loaded even though no import in the code has loaded it.
Contrary to what many people will wrongly tell you, it is almost never required to manipulate sys.path; there are popular Python projects on GitHub, running hundreds of thousands of lines of code, which either do not use it at all or use it only once in an ancillary role (perhaps because of a special requirement for a documentation tool). Just don't do it.
Also contrary to what many people will wrongly tell you, __init__.py files are not required to create packages in Python. They are simply a place where additional code can be placed for package initialization - for example, to create aliases for sub-package contents, or to limit what will be imported with a *-import (by setting __all__).
import .bb
This is just invalid. Relative imports only use the from syntax. See above for the correct syntax.

Suppose the same file structure as stated in OP, and:
main.py:
import module_a.aa
module_a.aa.thisFile()
module_a.aa.module_a.bb.thisFile()
aa.py:
import module_a.bb
def thisFile():
print("aa")
bb.py:
def thisFile():
print("bb")
Then, this will print
aa
bb
like you would expect. The main difference here, is that bb.py is imported in aa.py via module_a.bb. Running main.py gives no problem, however, running aa.py does not work in this way. That is why you might want to add folders to your path, such that you can call a function from a different file without this trouble. This is done via:
import os, inspect, sys
current_folder = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parent_folder = os.path.dirname(current_folder)
sys.path.insert(0,parent_folder)
Then you can import your files such as import file. If you consider this option, I would suggest to do some reading about how this works. Tip: make sure you avoid cyclic import problems.

Why can I import successfully without __init__.py?

What exactly is the use of __init__.py? Yes, I know this file makes a directory into an importable package. However, consider the following example:
project/
foo/
__init__.py
a.py
bar/
b.py
If I want to import a into b, I have to add following statement:
sys.path.append('/path_to_foo')
import foo.a
This will run successfully with or without __init__.py. However, if there is not an sys.path.append statement, a "no module" error will occur, with or without __init__.py. This makes it seem lik eonly the system path matters, and that __init__.py does not have any effect.
Why would this import work without __init__.py?

__init__.py has nothing to do with whether Python can find your package. You've run your code in such a way that your package isn't on the search path by default, but if you had run it differently or configured your PYTHONPATH differently, the sys.path.append would have been unnecessary.
__init__.py used to be necessary to create a package, and in most cases, you should still provide it. Since Python 3.3, though, a folder without an __init__.py can be considered part of an implicit namespace package, a feature for splitting a package across multiple directories.
During import processing, the import machinery will continue to
iterate over each directory in the parent path as it does in Python
3.2. While looking for a module or package named "foo", for each directory in the parent path:
If <directory>/foo/__init__.py is found, a regular package is imported and returned.
If not, but <directory>/foo.{py,pyc,so,pyd} is found, a module is imported and returned. The exact list of extension varies by platform
and whether the -O flag is specified. The list here is
representative.
If not, but <directory>/foo is found and is a directory, it is recorded and the scan continues with the next directory in the parent
path.
Otherwise the scan continues with the next directory in the parent path.
If the scan completes without returning a module or package, and at
least one directory was recorded, then a namespace package is created.

If you really want to avoid __init__.py for some reason, you don't sys.path. Rather, create a module object and set its __path__ to a list of directories.

if I want to import a into b, I have to add following statement:
No! You'd just say: import foo.a. All this is provided you run the entire package at once using python -m main.module where main.module is the entry point to your entire application. It imports all other modules, and the modules that import more modules will try to look for them from the root of this project. For instance, foo.bar.c will import as foo.bar.b
Then it seems that only the system path matters and init.py does not have any effect.
You need to modify sys.path only when you are importing modules from locations that are not in your project, or the places where python looks for libraries. __init__.py not only makes a folder look like a package, it also does a few more things like "export" objects to outside world (__all__)

When you import something it has to either:
Retrieve an already loaded module or
Load the module that was imported
When you do import foo and python finds a folder called foo in a folder on your sys.path then it will look in that folder for an __init__.py to be considered the top level module.
(Note that if the package is not on your sys.path then you would need to append it's location to be able to import it.)
If that is not present it will look for a __init__.pyc version possibly in the __pycache__ folder, if that is also missing then that folder foo is not considered a loadable python package. If no other options for foo are found then an ImportError is raised.
If you try deleting the __init__.pyc file as well you will see that the the initializer script for a package is indeed necessary.

Python 2.7 error in import

I know a lot of questions have been posted about the intra-package importing. I want to know whether the below is the way for Python 2.7 too.
Source/
anomalyCheck/
__init__.py
DLthput.py
ULPowerStats.py
ULThput.py
config/
__init__.py
configure.py
parserTools/
__init__.py
logParser.py
utilities/
__init__.py
plotLogResults.py
__init__.py
lteDebugger.py
---- lteDebugger.py----
import parserTools.logParser
import anomalyCheck.DLthput
import utilities.plotLogResults
import configuration.TDDFDDconfiguration
import anomalyCheck.ULthput
### Work done after here ####
------DLThput.py------
from ..utilities.plotLogResults import *
from ..parserTools.logParser import *
### Work done after here ####
------ULThput.py-------
from ..parserTools.logParser import *
from ..utilities.plotLogResults import *
Error :
Upon running the lteDebugger.py file, the error is
ValueError: Attempted relative import beyond toplevel package
File "C:\Users\manojtut\Desktop\Project_LTE_SVN\Source\lteDebugger.py", line 2, in
import anomalyChecker.DLthput
File "C:\Users\manojtut\Desktop\Project_LTE_SVN\Source\anomalyChecker\DLthput.py", line 1, in
I've read almost all available docs and Guido's guide for intra-package importing. Also, I guess I've everything exactly in the right place. Am I missing something here? Please point out. Thanks a lot in advance. :) :)
Edit 1: The Problem mentioned is solved by Amber's answer. So, lteDebugger.py is working fine by importing all other modules. Now, another problem is that, I am unable to solve is that... when I want to compile/interpret(whatever u want to call) the DLThput.py/ULthput.py , it is showing the same error as above ... ValueError: Attempted relative import beyond toplevel package. Do I have any solution other adding paths to sys hacks? I really don't want to do that unless it's the only thing to do.
So, how can I dodge this?

You're running lteDebugger.py, which means that any "packages" must be at least one level lower in the directory tree - they need to be contained inside a folder for Python to recognize them as packages rather than modules (and thus for relative imports to work).
anomalyCheck is recognized as a package, but its parent directory is not (because that's where lteDebugger.py is), and thus you aren't allowed to use relative imports to go up to that parent directory.
One way you could fix this is by moving everything except lteDebugger.py into a subdirectory, e.g.:
Source/
debugger/
anomalyCheck/
__init__.py
DLthput.py
ULPowerStats.py
ULThput.py
config/
__init__.py
configure.py
parserTools/
__init__.py
logParser.py
utilities/
__init__.py
plotLogResults.py
__init__.py
lteDebugger.py
and then lteDebugger.py would do things like import debugger.anomalyCheck.DLthput.py.

Multilevel relative import

Multilevel relative import
I have following folder structure
top\
__init__.py
util\
__init__.py
utiltest.py
foo\
__init__.py
foo.py
bar\
__init__.py
foobar.py
I want to access from foobar.py the module utiltest.py. I tried following relative import, but this doesn't work:
from ...util.utiltest import *
I always get
ValueError: Attempted relative import beyond toplevel package
How to do such a multileve relative import?

I realize this is an old question, but I feel the accepted answer likely misses the main issue with the questioner's code. It's not wrong, strictly speaking, but it gives a suggestion that only coincidentally happens to work around the real issue.
That real issue is that the foobar.py file in top\foo\bar is being run as a script. When a (correct!) relative import is attempted, it fails because the Python interpreter doesn't understand the package structure.
The best fix for this is to run foobar.py not by filename, but instead to use the -m flag to the interpreter to tell it to run the top.foo.bar.foobar module. This way Python will know the main module it's loading is in a package, and it will know exactly where the relative import is referring.

You must import foobar from the parent folder of top:
import top.foo.bar.foobar
This tells Python that top is the top level package. Relative imports are possible only inside a package.

Import paths - the right way?

I know there are A LOT of similar or the same questions, but i still cannot understand / find the right way for me to work with modules. Python is my favorite language, and i like everything in it except working with imports: recursive imports (when you try to reference a name that is not yet there), import paths, etc.
So, I have this kind of a project structure:
my_project/
package1/
__init__.py
module1
module2
package2/
__init__.py
module1
module2
Package1 may be used a standalone unit, but is also expected to be imported by package2.
What am i doing now, is that, for example, in package1.module1 i write from package1 import module2, i.e. using full path to imported module. I do this because if i use import module2 -- this will not work when the module will be imported from another package (package2). I also cannot use from . import module2 -- this will not work when running module1 directly.
OK, so for from package1 import module2 in package1.module1 to work in both cases (when running directly package1.module1 and when importing it from package2) i add these lines at the beginning of package1.module1:
import os, sys
currDir = os.path.dirname(os.path.realpath(__file__))
rootDir = os.path.abspath(os.path.join(currDir, '..'))
if rootDir not in sys.path: # add parent dir to paths
sys.path.append(rootDir)
For me this works, but i feel this is not pythonic. Am i doing something wrong?
Should i, instead, always run package1.module1 from project root? If so, this makes inconvenient to run it from an IDE -- i need somehow to set paths in it.
UPDATE: I tried to add a file root.pth to package1 dir with contents of ... But it didn't work -- i guess it's intended for something else.
CONCLUSIONS:
Always use absolute imports: import package1.module1
Add a bootstrapper to the root folder to start some of the modules as a standalone script. This solves running the script form an IDE and is a pythonic approach.
On 4/22/07, Brett Cannon wrote:
This PEP is to change the if __name__ == "__main__": ... idiom to
if __name__ == sys.main: ... so that you at least have a chance
to execute module in a package that use relative imports.
Ran this PEP past python-ideas. Stopped the discussion there when too
many new ideas were being proposed. =) I have listed all of them in
the Rejected Ideas section, although if overwhelming support for one
comes forward the PEP can shift to one of them.
I'm -1 on this and on any other proposed twiddlings of the __main__
machinery. The only use case seems to be running scripts that happen
to be living inside a module's directory, which I've always seen as an
antipattern. To make me change my mind you'd have to convince me that
it isn't.
--Guido van Rossum

What is the entry point for your program? Usually the entry point for a program will be at the root of the project. Since it is at the root, all the modules within the root will be importable, provided there is an __init__.py file in them.
So, using your example:
my_project/
main.py
package1/
__init__.py
module1
module2
package2/
__init__.py
module1
module2
main.py would be the entry point for your program. Because the file that is executed as main is automatically put on the PYTHONPATH, both package1 and package2 are available from the top level import.
# in main.py
from package1.module1 import *
from package1.module2 import *
# in package1.module1
import module2
from package2.module1 import *
# in package2.module1 import *
import module2
from package1.module1 import *
Note that in the above, package1 and package2 depend on each other. That should never be the case. But this is just an example of being able to import from anywhere.
main.py doesn't have to be anything fancy either. It can be very simple:
# main.py
if __name__ == '__main__':
from package1.module1 import SomeClass
SomeClass().start()
The point I'm trying to make, is that if a module needs to be accessible by other modules, that module should be available as a top level import. A module should not attempt to put itself as a top level import (directly on the PYTHONPATH).
It should be the responsibility of the project for ensuring that all imports can be satisfied if the module is included directly in the project. There are two ways to do this. The first is by creating a bootstrapper file such as main.py in the project folder. The other, is by creating a file that adds all relevant paths to PYTHONPATH, that is loaded by any entry points that may exist.
For example:
# setup.py
import sys
def load():
paths = ['/path1/','/path2/','/path3/']
for p in path:
sys.path.insert(0, p)
# entrypoint.py
from setup import load
load()
# continue with program
The main thing to take away, is that a module is not supposed to put itself on the path. The path should be determined automatically by the entry point into the program, or defined explicitly by a setup script that knows where all the relevant modules are.

I generally create each package as an installable package (ie, create a setup.py file), and then install them into a virtualenv just for this project, using pip.
You can even install the using pip -e if they are still under development.

I'm 5 years late here.. but just do export PYTHONPATH=/path1:/path2: (notice trailing ":") - this way your working directory (from which you run python) will be in the path.