Develop Reference

Why does error occurs when importing from same *sub*directory?

Consider this folder structure:
main.py
module_a/
aa.py
bb.py
__init__.py
In main.py, I import aa as:
from module_a import aa
aa.yyy()
Then in aa.py, I import bb and include its functions as:
import bb
bb.xxx()
However, when I run main.py, python says "No module named 'bb'".
May I know why this happens. What is the correct way to import bb.
Thanks!!!
I have tried to write aa.py as:
import .bb
bb.xxx()
But it still does not work.

why this happens
Because the aa folder is not a place that Python is searching for modules.
Python's imports are absolute by default. They only look in specific places determined by sys.path. In main.py, import module_a.aa works because the root folder of the project happens to be on the sys.path; that folder contains a module_a folder; and that folder contains aa.py.
What is the correct way to import bb.
Please use relative imports between files in your package. In this case, the necessary import in aa.py looks like:
from . import bb
Absolute imports are error-prone; a project that uses two packages whose contents have overlapping names will run into namespace collisions. (Sadly, the standard library uses absolute imports in most places, such that projects need to ban certain module names for safety.) Relative imports will also require much less maintenance, should you later rename a sub-package.
The only thing relative imports require is that the package gets loaded, which typically will happen automatically with the first (yes, absolute) import of any of the package contents. When the package is loaded, it automatically sets a __package__ attribute on the modules in that package, which Python can use to resolve the relative imports. It's important to note that relative imports are relative to the package hierarchy, not the directory structure, which is why this is necessary; imports like from .. import example do not work by figuring out the current file location and then going up a level in the directory hierarchy. Instead, they check the __package__ to figure out what the containing package is, then check the file/folder location for that, and work from there.
If the "driver" script is within the package, run it as a module, using the -m switch for Python. For example, from the root folder, if module_a/aa.py is the driver, use python -m module_a.aa. This instructs Python that module_a is the containing package for aa.py, and ensures it gets loaded even though no import in the code has loaded it.
Contrary to what many people will wrongly tell you, it is almost never required to manipulate sys.path; there are popular Python projects on GitHub, running hundreds of thousands of lines of code, which either do not use it at all or use it only once in an ancillary role (perhaps because of a special requirement for a documentation tool). Just don't do it.
Also contrary to what many people will wrongly tell you, __init__.py files are not required to create packages in Python. They are simply a place where additional code can be placed for package initialization - for example, to create aliases for sub-package contents, or to limit what will be imported with a *-import (by setting __all__).
import .bb
This is just invalid. Relative imports only use the from syntax. See above for the correct syntax.

Suppose the same file structure as stated in OP, and:
main.py:
import module_a.aa
module_a.aa.thisFile()
module_a.aa.module_a.bb.thisFile()
aa.py:
import module_a.bb
def thisFile():
print("aa")
bb.py:
def thisFile():
print("bb")
Then, this will print
aa
bb
like you would expect. The main difference here, is that bb.py is imported in aa.py via module_a.bb. Running main.py gives no problem, however, running aa.py does not work in this way. That is why you might want to add folders to your path, such that you can call a function from a different file without this trouble. This is done via:
import os, inspect, sys
current_folder = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parent_folder = os.path.dirname(current_folder)
sys.path.insert(0,parent_folder)
Then you can import your files such as import file. If you consider this option, I would suggest to do some reading about how this works. Tip: make sure you avoid cyclic import problems.

Classes importing other classes errors

I have three scripts:
C:\code\voiceTerm\master.py:
from voice_terminal_module.voice_terminal import VoiceTerminal
vterm = VoiceTerminal()
C:\code\voiceTerm\voice_terminal_module\voice_terminal.py:
from chatbot_module.chatbot_module import Chatbot
class VoiceTerminal:
print("INITIALIZING VOICE TERMINAL")
cb = Chatbot()
C:\code\voiceTerm\voice_terminal_module\chatbot_module\chatbot_module.py:
class Chatbot:
print("CHATBOT INITIALIZED")
Here is the wierd thing: When I run chatbot_module.py it works, and if I run voice_terminal.py it works. For some reason however, master.py errors out with the following message:
Traceback (most recent call last):
File "c:\code\voiceTerm\master.py", line 1, in <module>
from voice_terminal_module.voice_terminal import VoiceTerminal
File "c:\code\voiceTerm\voice_terminal_module\voice_terminal.py", line 1, in <module>
from chatbot_module.chatbot_module import Chatbot
ModuleNotFoundError: No module named 'chatbot_module'
Why does it work sometimes, but sometimes not?

You'll need to restructure your project in a way that you have either: (1) two separate and installed packages containing each module or; (2) one package containing all modules. The simplest way forward is (2) and it looks like:
master.py
src/ # Name it anything you wish
__init__.py
voice_terminal.py
chatbot_module.py
Then use relative imports in voice_terminal.py
from .chatbot_module import Chatbot
Rule: Packages does not know others exists unless the other package is installed, added to sys.path, a subpackage, or part of a parent package.

Python general importing guide (for your own code)
I'm sure there should be a duplicate for this by now, but I can't find it, so I'm writing a full essay.
To solve the problem properly, you must take care of two things:
Make sure that sys.path - the path that Python searches for modules - includes the path to the project root.
Make sure that the import statements in the code are written to work with that path.
The search path
For the first part, you must understand how sys.path works. It is a list of folders where Python will look, that is set up automatically at startup (even if you don't import sys - just like sys.argv is).
Normally, it will be a list which contains, in order:
A path to the entry point for execution (the details of this will depend on how Python is started; typically this is '' when running a .py file directly from the same directory, a path the another directory if you do e.g. python somewhere/else/file.py, and an explicit path when using the -m switch)
Paths to system libraries
Paths to virtual environment libraries, if a virtual environment is active
Paths to things that were explicitly installed by the user
You can modify this list, with the expected impact on future import statements; but you ordinarily should not.
To ensure your project root is on the path, normally you should just install the project - ideally into a virtual environment. Please see the Python packaging guide for details. Failing that, make sure to start the project from just outside the folder containing your "top-level" code. In OP's case, that means C:\code. This will ensure that C:\code (or something equivalent) is on sys.path, which we can then rely upon for the second step.
The imports
We can fundamentally do this in two ways: By absolute imports specifying the path from the root, or by relative imports specifying the path from the current source file. Read more on Stack Overflow about relative imports here and here.
Absolute imports
In either case, we want to treat the root folder for our project (here, C:\code\voiceTerm) as a package. For absolute imports, this means we will always mention the root folder name in our import path.
Thus:
In C:\code\voiceTerm\master.py: from voiceTerm.voice_terminal_module.voice_terminal import VoiceTerminal
In C:\code\voiceTerm\voice_terminal_module\voice_terminal.py: from voiceTerm.voice_terminal_module.chatbot_module.chatbot_module import Chatbot
(You don't really want to have _module in your folder or file names. It doesn't really add information, makes this considerably harder to type, and is actually a bit misleading.)
We can also import entire modules: import voiceTerm.master, import voiceTerm.voice_terminal_module.voice_terminal; import voiceTerm.voice_terminal_module.chatbot_module.chatbot_module. Additionally, by naming a file __init__.py, we become able to import the folder as a package: import voiceTerm; import voiceTerm.voice_terminal_module; import voiceTerm.voice_terminal_module.chatbot_module. We can also import a module from one of those packages: from voiceTerm import master; from voiceTerm.voice_terminal_module import voice_terminal; from voiceTerm.voice_terminal_module.chatbot_module import chatbot_module.
Relative imports
With relative imports, we must use the from syntax, and we can only import things from within our own package hierarchy. However, we have the advantage that we don't have to specify full paths, and we can rename packages without having to edit the code. My personal recommendation is to use relative imports where possible; this also makes a strong visual distinction with imports from system libraries or other third-party packages.
For a relative import, first we specify the package or sub-package that we're importing from, using a relative import path. By starting with a single ., we start from the current directory, looking for another module or package within the same folder.
Thus:
In C:\code\voiceTerm\master.py: from .voice_terminal_module.voice_terminal import VoiceTerminal to import the class; from .voice_terminal_module import voice_terminal to import the module.
In C:\code\voiceTerm\voice_terminal_module\voice_terminal.py: from .chatbot_module.chatbot_module import Chatbot to import the class; from .chatbot_module import chatbot_module to import the module.
Additional .s navigate up the package hierarchy (but we cannot go beyond the root this way). For example, from ... import master to import the top-level master.py from the lower-down chatbot_module.py. (Of course, this wouldn't actually work in this case, because it would be a circular import.) To import another source file (as a module) from the same directory, simply from . import other_file. You get the idea.

I solved it. You can appearantly use importlib for this. I replaced the import code in my voice_terminal.py script with this:
chatbot_module = SourceFileLoader("chatbot_module", "C:/code/voiceTerm/voice_terminal_module/chatbot_module/chatbot_module.py").load_module()

Importing file from another imported file

I have the following directory structure:
base_folder
methods_folder
method_1.py
method_2.py
.
.
.
method_n.py
class_methods.py
top_class.py
class_methods.py imports the other files in the same directory, like this:
from method_1 import method_1
from method_2 import method_2
.
.
.
from method_n import method_n
(obs: these methods files has a method with its own file names inside them)
If I run class_methods.py by myself, no problem. But if I try to run top_class.py, which imports class_methods.py, I get the error no module named method_1
So, when executing top_class.py, it is not seeing the files in methods_folder/. Why?

the correct import inside top_class.py would be from methods_folder.method_n import method_n. This is because you are treating methods_folder as a package. If you are running a version of Python that is before 3.3 you must also unclude __init__.py file inside the methods_folder in order to turn it into a package.

Files only have direct access to things they import. Say we have a.py which imports b.py, and b.py imports c.py. When running within a.py functions in b.py that use c.py, this will work fine, because a has access to b, and b has access to c. This does not mean, however, that the imports chain (as in C++) and you can use functions from c in a. You will get an error, because a can only see the contents of b, which it imported.
So if you want to use all your method_i.py files from within top_class.py, you need to import them directly in the same file.
Edit: You also have some other issues. To import other files in a subfolder, you would need to, inside top_class.py, call import methods_folder.method_i. To import something in the same directory, just use import method_i. Since you have a method of the same name in each file, what you have works fine in class_methods.py You also need to create an empty file called __init__.py in any folder containing python files you intend to import to/from which lets python know it's allowed to look there.

You can create an importable package in python in one of two ways. The first way is what you are doing: you create a file called my_package.py and import it with import my_package. This is commonly used for simpler packages that don't need to be further broken up. For this to work, your .py file has to be on the PYTHONPATH which is an environment variable that tells python where to look for packages. If not defined, there are some default places that python will use to look for packages. One of these default locations is the current working directory, which is why your first set of imports works.
In theory you should be able to run the second piece of code from the same location (python ../top_class.py) and use the same import style, but I assume you are changing directories to run that file. This means your files are no longer in the current working directory and no longer findable by python.
One way to get your code to work using the existing style would be to define PYTHONPATH with the location of your methodX.py files. You will typically add to the python search path like this:
PYTHONPATH=$PYTHONPATH:./methods_folder python top_class.py
This tells python to look in methods_folder, in addition to the standard places, when you try to import something. Playing with the PYTHONPATH gets a little annoying after a while, so I actually prefer the next approach.
A second way to create a package is by creating a folder with an __init__.py file inside. This tells python that you want it to treat that directory as a package. This is the preferred style for more complicated pieces of code that might benefit from organization across multiple files. For your example, you could organize your code in the following way:
base_folder
methods_folder
__init__.py
method_1.py
method_2.py
.
.
.
method_n.py
class_methods.py
top_class.py
And then your import in top_class.py would look like this:
from methods_folder.method1 import method1
from methods_folder.method2 import method2
from methods_folder.method3 import method3
This has the effect of creating a top level methods_folder package with modules method1, method2, etc. Because methods_folder is in the same directory as the one you are running top_class.py from, python picks that up as a package using the default PYTHONPATH and lets you import from within it.

I assume you are running them from their respective directories? Unless they are installed in your Python path (I'm going to assume they are not), then Python will by default look for imports in your current directory. So when you run class_methods.py from its directory, then it can find methods_1.py in order to satisfy from methods_1 import method_1. But when you execute top_class.py, it looks for methods_1.py or methods_1/__init__.py, neither of which it fines from that directory.
Assuming Python 3, you would need to use relative imports in class_methods.py.
# class_methods.py
from .methods_1 import method_1
from .methods_2 import method_2
This will let you run it from top_class.py. Unfortunately, you can't use relative imports when running a script in the same package, so you wouldn't be able to run class_methods.py directly.
Another option in that case is to keep the absolute imports in class_methods.py and add the methods folder to the path in top_class.py.
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), 'methods'))
import class_methods
from methods_1 import method
Of course editing sys.path is fine for small scripts and standalone things. But if this grows past being a script, you'll need to be careful about doing it, and you'll probably just want to come up with another solution. Likely, the best thing is to create a package that you install (you can still do this from your source directory while you are developing) and then you can import the same way from anywhere.

The recommended way of running a python script is using the -m switch from the parent of your root package - so in your case:
$ cd base_folder
$ python -m top_class
Python will automatically add the base_folder to its sys.path and you don't need to do any sys.path/PYTHOPATH hacks that are just this - hacks that bloat the code with boilerplate and will blow when least expected.
Now to run the class_methods the correct way is also
$ cd base_folder
$ python -m methods_folder.class_methods
but then the imports in class_methods should be modified to either absolute:
from methods_folder.method1 import method1
...
or relative:
from .method1 import method1
...

Import a library which is in a sibling of the current folder

With the folder structure
lib/
abcd/
__init.py__
lib.py
app.py
the code
from lib.abcd import lib
works. But with this file structure:
bin/
app.py
lib/
abcd/
__init.py__
lib.py
the code
from ..lib.abcd import lib
gives an import error.
How to do the import properly when the library is in a sibling of the current folder? (or subfolder of a sibling folder)
I know that there might some hack that involves adding lib/ to the PATH, but is there an elegant Pythonic solution?
If not, is there a real internal reason to prevent users to do this simple import in a simple way?

Methods to do this
Method #1: Using the sys module: You can easily accomplish what you are trying to do using the sys module. To import the lib package, you can use one of the two codes listed below:
import sys
sys.path.append('<PATH_TO_LIB_FOLDER>')
from lib.abcd import lib
or
import sys
sys.path.insert(0, '<PATH_TO_LIB_FOLDER>')
Method #2: Using the os module: Another method is to use the os module. Here is an example code that imports the lib module using the os module by invoking the os.path.join method:
import os
path = os.path.join("<PATH>/lib/abcd", "lib")
from lib.abcd import lib
Method #3: Add the module to your PYTHONPATH: This is not the best method in most cases, but if you don't want to keep using the sys or os module to import lib, this is ideal. All you have to do is type this in your bash terminal:
export PYTHONPATH=<PATH_TO_LIB> python lib.py
Then in your python shell you can import it like this:
from lib.abcd import lib
Method #4: Combine the sys and os module (recommended): This is the most efficient method and will save you a lot of time. This code combines the os and sys module like this:
import sys, os
sys.path.append(os.path.abspath(os.path.join('..', 'lib')))
Then you can import your module with ease like this:
from lib.abcd import lib
How all the codes work:
All the codes above are very simple. All the examples except for "Method #3", add your module to the PYTHONPATH temporarily. "Method #3" on the other hand, adds the module to your PYTHONPATH permanently.

Surface level look
Normally, you can't. When importing a file, Python only searches the current directory, the directory that the entry-point script is running from, and sys.path which includes locations such as the package installation directory (it's actually a little more complex than this, but this covers most cases).
The reason you don't see this problem for importing installed modules is because they are installed in a location that is already on your path, or the location is added to the path by the installing utility (pip for example).
You can add to the Python path at runtime:
import sys
sys.path.insert(0, '../lib')
import file
You can also use sys.path.append('../lib'), but then it will be searched for last in your path and may be overridden by previous path entries.
I have read through the import documentation extensively, and as far as I can tell, the answer to your question is no, there is no way to do this in a purely "Pythonic" way.
More in depth look:
Looking deeper into the import documentation explains this:
The import statement combines two operations; it searches for the named module, then it binds the results of that search to a name in the local scope. The search operation of the import statement is defined as a call to the __import__() function, with the appropriate arguments. The return value of __import__() is used to perform the name binding operation of the import statement.
Looking more closely at __import__:
__import__(name, globals=None, locals=None, fromlist=(), level=0)
Note: This is an advanced function that is not needed in everyday Python programming, unlike importlib.import_module().
This function is invoked by the import statement. It can be replaced (by importing the builtins module and assigning to builtins.__import__) in order to change semantics of the import statement, but doing so is strongly discouraged as it is usually simpler to use import hooks (see PEP 302) to attain the same goals and does not cause issues with code which assumes the default import implementation is in use. Direct use of __import__() is also discouraged in favor of importlib.import_module().
level specifies whether to use absolute or relative imports. 0 (the default) means only perform absolute imports. Positive values for level indicate the number of parent directories to search relative to the directory of the module calling __import__() (see PEP 328 for the details).
This makes me think that you can specify a level in some way that may make the import automatically look on the parent path. I'm guessing that when import is called from your app.py that is not in it's own directory, level is set to 0 and it searches the same level and deeper.
When you call import in app.py from a subfolder, level is still set to 0 and thus can't find the other directories above it. I am investigating a "Pythonic" way of setting this level to 1 when running your script which would appear to fix this problem.

Python - fails importing package

I have trouble importing package.
My file structure is like this:
filelib/
__init__.py
converters/
__init__.py
cmp2locus.py
modelmaker/
__init__.py
command_file.py
In module command_file.py I have a class named CommandFile which i want to call in the cmp2locus.py module.
I have tried the following in cmp2locus.py module:
import filelib.modelmaker.command_file
import modelmaker.command_file
from filelib.modelmaker.command_file import CommandFile
All these options return ImportError: No modules named ...
Appreciate any hint on solving this. I do not understand why this import does not work.

To perform these imports you have 3 options, I'll list them in the order I'd prefer. (For all of these options I will be assuming python 3)
Relative imports
Your file structure looks like a proper package file structure so this should work however anyone else trying this option should note that it requires you to be in a package; this won't work for some random script.
You'll also need to run the script doing the importing from outside the package, for example by importing it and running it from there rather than just running the cmp2locus.py script directly
Then you'll need to change your imports to be relative by using ..
So:
import filelib.modelmaker.command_file
becomes
from ..modelmaker import command_file
The .. refers to the parent folder (like the hidden file in file systems).
Also note you have to use the from import syntax because names starting with .. aren't valid identifiers in python. However you can of course import it as whatever you'd like using from import as.
See also the PEP
Absolute imports
If you place your package in site-packages (the directories returned by site.getsitepackages()) you will be able to use the format of imports that you were trying to use in the question. Note that this requires any users of your package to install it there too so this isn't ideal (although they probably would, relying on it is bad).
Modifying the python path
As Meera answered you can also directly modify the python path by using sys.
I dislike this option personally as it feels very 'hacky' but I've been told it can be useful as it gives you precise control of what you can import.

To import from another folder, you have to append that path of the folder to sys.path:
import sys
sys.path.append('path/filelib/modelmaker')
import command_file