Develop Reference

Unable to import from root directory to subdirectory without modifying sys.path (Python 3.8)

What I have tried
First let me say, I have read maybe 5 or 6 other stackoverflow questions related to this and many things have been said about this. Here's what I found and what I tried:
"Do sys.path.append(<root dir>) before other imports."
Issue: This works, but it's against PEP standard practice so I'd rather find a solution that does not require modifying sys.path.
"Use the imp library to find your module."
Issue: The solution outlined in that answer is deprecated as importlib is the new imp, and I'm not sure how to use importlib even after looking at the docs.
"Add an empty __init__.py file in your root directory."
Issue: This does not work.
Details of my problem
Here is my file structure:
Project/
├ subdir/
| └ script.py
├ __init__.py
└ module.py
(Note that I left the empty __init__.py file in there just in case it's needed.)
I'm trying to import from module.py to subdir/script.py:
Contents of module.py:
def func():
print('func was called')
Contents of subdir/script.py:
from module import func
func()
However, when I run python3 script.py in terminal (on Mac), I get this error message:
Traceback (most recent call last):
File "script.py", line 1, in <module>
from module import func
ModuleNotFoundError: No module named 'module'
I would appreciate a straightforward solution that does not require modifying sys.path.

Probably the easiest way to solve your problem is to restructure your project:
Project
├── script.py
└── module
├── __init__.py
└── module.py

I solved my issue by setting PYTHONPATH. In my case, it works with this.
PYTHONPATH=<root dir> python3 subdir/script.py

ModuleNotFoundError: No module named 'sib1'

I have a project hierarchy like below, when I run python src/bot/main I got no error. While if I run python -m src.bot.main I got error. Why?
This is my file hierarchy:
MyProject
└── src
├── __init__.py
├── bot
│   ├── __init__.py
│   ├── main.py
│   └── sib1.py
└── mod
├── __init__.py
└── module1.py
This is the content of main.py:
import sys
if __name__ == "__main__":
# sys.path will print the parent folder.
print(sys.path, end="\n\n")
# my problem is on this line.
import sib1
sib1.test()
The error:
Traceback (most recent call last):
File "/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/me/Desktop/test_py/src/bot/main.py", line 16, in <module>
import sib1
ModuleNotFoundError: No module named 'sib1'
Some conclusion I've made so far:
Since the output of sys.path in both cases include /Users/me/Desktop/MyProject, the reason should not related to scope?
The output of sys.path of both python -m src.bot.main and python src/bot/main:
(test) ✔ me Desktop/test_py % python -m src.bot.main
['/Users/me/Desktop/test_py', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python39.zip', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/lib-dynload', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/site-packages']

I will try my best to clarify each of my confusions in a Q&A form, and organize #Brain's comments and during the way:
Q1: when I run python src/bot/main I got no error.
The sys.path will include the current directory containing the file main.py, i.e. the interpreter will see the file MyProject/src/bot:
import sib1
is (logically) equivalent to:
import /path/to/MyProject/src/bot/sib1.py
Hence, no error.
BUT this is called IMPLICIT-RELATIVE-IMPORT, which should be used only on the main script. See Example.
Q2: While if I run python -m src.bot.main I got error. Why?
Now it's time to quote #Brain's valuable (first) comment:
Using python -m src.bot.main tells Python that src is a top-level package. Everything below src in the directory structure will be considered submodules/subpackages of src. The proper name for sib1 under that organization is src.bot.sib1. No top-level module named sib1 exists as far as Python is concerned.
(emphasis by me)
So:
The -m option will provide some context information(details, I recommend reading the first two high-upvoted answers entirely) for the file you're going to run.
Now every absolute (import-)path that does not begin with src. will be regarded as built-in or third-party libraries installed in your virtual environment. Since I didn't install any package called sib1, you got the error. (Ignore about the inclusion of the current directory in the sys.path. I guess this is for compatibility reason. Remember that we DONT want IMPLICIT-RELATIVE-IMPORT in python3)
You can use relative path BUT you should not go out of the src (since src is the top-level package in the execution) by prepending too many .'s.
For example this will work:
from . import sib1
from ..mod import module1 # The `..` is equivalent to the MyProject/src.
module1.hello()
sib1.test()
Finally, don't test your package by inserting many if __name__ == '__main__'. Do so by professional tools:
If you only need to run submodules for testing purpose, you could consider using more robust testing tools like doctest (lightweight) or unittest (heavyweight).

PyPi-installed app fails with ModuleNotFound

I created a project that runs fine locally, but fails when run from PyPi deploy/install, with ModuleNotFound. The link contains a very small sample, and run / deploy instructions. My original thought was regarding inheritance, but it seems more basic.
Traceback (most recent call last):
File "/Users/val/python/vscode/inheritance/venv/bin/inheritance-run", line 5, in <module>
from inheritance.run import start
ModuleNotFoundError: No module named 'inheritance'

Your setup.py has:
packages=find_packages(),
This will find modules (directories with __init__.py files) in the same directory as your setup.py file, but you don't have any. Instead, you have a base.py file.
If you want to be able to import inheritance, you should move this into an __init__.py file in new directory named inheritance, so you have something like:
.
├── inheritance
│   └── __init__.py
└── setup.py
Then setuptools will find this module and include it as an importable package in your project.
FYI, It's usually preferable to use a src-based layout instead, however. See https://github.com/pypa/sampleproject/ for a full example.

Python - ModuleNotFoundError: No module named

I'm new in Python and I'm having the following error with this simple example:
This is my project structure:
python_project
.
├── lib
│   ├── __init__.py
│   └── my_custom_lib.py
└── src
├── __init__.py
└── main.py
And this is the error when I execute the src/main.py file:
☁ python_project python src/main.py
Traceback (most recent call last):
File "src/main.py", line 3, in <module>
from lib import my_custom_lib
ImportError: No module named lib
If I move the main.py file to the root and then I execute this file again, works... but is not working inside src/ directory
This is my main.py:
from lib import my_custom_lib
def do_something(message):
my_custom_lib.show(message)
do_something('Hello World!')
Note: When I execute the same code from Pycharm is working fine, but not from my terminal.

Your PYTHONPATH is set to the parent directory of the executed script. So if the executed script is inside a directory src, it will never be able to find the sister directory lib because it isn't within the path. There's a few choices;
Just move lib/ into src/ if it belongs to your code. If it's an external package, it should be pip installed.
Have a top-level script outside of src/ that imports and runs src.main. This will add the top-level directory to python path.
In src/main.py modify sys.path to include the top-level directory. This is usually frowned upon.
Invoke src/main.py as a module with python -m src.main which will add the top-level directory to the python path. Kind of annoying to type, plus you'll need to change all your imports.

If I may add to MarkM's answer, if you wanted to keep your current directory structure and still make it work, you could add a setup.py in your root dir, where you can use setuptools to create a package you could install.
If your file had something along the lines of:
# setup.py
from setuptools import find_packages, setup
setup(
name='foo',
version=`1.0.0`,
packages=find_packages(),
entrypoints={
'console_scripts': [
'foo=src.main:main',
],
},
)
And then you do pip install [--user] -e path/to/directory you'll get an "editable package" which will effectively a symlink to the package in your development directory, so any changes you make will not require a reinstall (unless of course you rejig package structure or add/remove/edit entry points).
This does assume your src/main.py has a main function.
You'll also need __init__.py files in your "package" directories, even in Python3, as otherwise Python assumes these are namespace packages (Won't go into detail) and the find_packages() call won't find them.
This will also allow your relative imports to work. Absolute imports will only work when invoking the script from your entry point but not when calling the script directly in your development directory.

You should have your main.py script above all python packages in your directory structure. Try to update your project to the following structure:
.
|__ main.py
|__ lib
| |__ __init__.py
| |__ your_custom_lib.py
|__ another_python_package
|__ __init__.py
|__ another_python_scripts
After that, python main.py in your project directory will work.

You are using the from a import b incorrectly. it should look like this:
import lib.my_custom_lib
The other method is used to import certain methods, functions, and classes from a module, not the module itself. To import a specific function from the my_custom_lib module it would look like this:
from lib.my_custom_lib import foo

Try using a relative import instead:
from ..lib import my_custom_lib

in my case in visual code
actual error in line 1
I didn't imported _typeshed module but by default it was their
so delete that module if you found in line 1

MarkM's answer is still excellent; I'm using PyPi now. I attempted to disambiguate what he was saying about "current directory". In case my confusion was an intended feature, here is how I did it.
vagrant#testrunner:~/pypath$ tree
.
├── proga
│   └── script1.py
└── progb
└── script1.py
script1.py is the same in both directories:
#!/usr/bin/env python3
import sys
print(sys.path)
Where I run it from makes no difference, PYTHONPATH prepends the directory containing the script I specify:
vagrant#testrunner:~/pypath/proga$ ./script1.py
['/home/vagrant/pypath/proga', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/vagrant/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']
vagrant#testrunner:~/pypath/proga$ ../progb/script1.py
['/home/vagrant/pypath/progb', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/vagrant/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']

For me importing with explicit paths works best in such situations. If you need to go up in the tree use '..'. The plumbing is a bit cumbersome, but it always works.
path = os.path.abspath(os.path.join(pathlib.Path(__file__).parent.absolute(), '..', 'subdir', 'myFile.py'))
loader = importlib.machinery.SourceFileLoader('myFile', path)
spec = importlib.util.spec_from_loader('myFile', loader)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# now use the module:
module.myMethod()
myClassInstance = module.myClass()

Python Nose Import Error

I can't seem to get the nose testing framework to recognize modules beneath my test script in the file structure. I've set up the simplest example that demonstrates the problem. I'll explain it below.
Here's the the package file structure:
./__init__.py
./foo.py
./tests
./__init__.py
./test_foo.py
foo.py contains:
def dumb_true():
return True
tests/test_foo.py contains:
import foo
def test_foo():
assert foo.dumb_true()
Both init.py files are empty
If I run nosetests -vv in the main directory (where foo.py is), I get:
Failure: ImportError (No module named foo) ... ERROR
======================================================================
ERROR: Failure: ImportError (No module named foo)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python/site-packages/nose-0.11.1-py2.6.egg/nose/loader.py", line 379, in loadTestsFromName
addr.filename, addr.module)
File "/usr/lib/python/site-packages/nose-0.11.1-py2.6.egg/nose/importer.py", line 39, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/usr/lib/python/site-packages/nose-0.11.1-py2.6.egg/nose/importer.py", line 86, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/user/nose_testing/tests/test_foo.py", line 1, in <module>
import foo
ImportError: No module named foo
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (errors=1)
I get the same error when I run from inside the tests/ directory. According to the documentation and an example I found, nose is supposed to add all parent packages to the path as well as the directory from which it is called, but this doesn't seem to be happening in my case.
I'm running Ubuntu 8.04 with Python 2.6.2. I've built and installed nose manually (not with setup_tools) if that matters.

You've got an __init__.py in your top level directory. That makes it a package. If you remove it, your nosetests should work.
If you don't remove it, you'll have to change your import to import dir.foo, where dir is the name of your directory.

Are you in a virtualenv? In my case, nosetests was the one in /usr/bin/nosetests, which was using /usr/bin/python. The packages in the virtualenv definitely won't be in the system path. The following fixed this:
source myvirtualenv/activate
pip install nose
which nosetests
/home/me/myvirtualenv/bin/nosetests

To those of you finding this question later on: I get the import error if I don't have an __init__.py file in my tests directory.
My directory structure was like this:
./tests/
./test_some_random_stuff.py
If I ran nosetests:
nosetests -w tests
It would give the ImportError that everyone else is seeing. If I add a blank __init__.py file it works just fine:
./tests/
./__init__.py
./test_some_random_stuff.py

Another potential problem appears to be hyphens/dashes in the directory tree. I recently fixed a nose ImportError issue by renaming a directory from sub-dir to sub_dir.

Of course if you have a syntax error in the module being imported that will cause this. For me the problem reared its head when I had a backup of a tests file with a path like module/tests.bak.py in the same directory as tests.py. Also, to deal with the init package/module problem in a Django app, you can run the following (in a bash/OSX shell) to make sure you don't have any init.pyc files lying around:
find . -name '*.pyc' -delete

I got this error message because I run the nosetests command from the wrong directory.
Silly, but happens.

I just ran into one more thing that might cause this issue: naming of tests in the form testname.test.py. That extra . confounds nose and leads to it importing things it should not. I suppose it may be obvious that using unconventional test naming conventions will break things, but I thought it might be worth noting.

For example, with the following directory structure, if you want to run nosetests in m1, m2 or m3 to test some functions in n.py, you should use from m2.m3 import n in test.py.
m1
└── m2
├── __init__.py
└── m3
├── __init__.py
├── n.py
└── test
└── test.py

Just to complete the question:
If you're struggling with structure like this:
project
├── m1
├ ├── __init__.py
├ ├── foo1.py
├ └──m2
├ ├── __init__.py
├ └── foo2.py
├
└── test
├── __init__.py
└── test.py
And maybe you want to run test from a path outside the project,
include your project path inside your PYTHONPATH.
export PYTHONPATH=$PYTHONPATH:$HOME/path/to/project
paste it inside your .profile.
If you're under a virtual environment, paste it inside the activate in your venv root