Develop Reference

How to import Python module in another folder, without relative import and editing PYTHONPATH

This is my folder structure:
.
├── main.py
├── formats
│ ├── __init__.py
│ └── writer.py
└── misc
├── __init__.py
└── util.py
In main.py, I can import util.py using:
from misc.util import sth
However, I can't import util.py in writer.py, using the above statement, and this command:
python formats/writer.py
Now the simplest solution is to mess with the PYTHONPATH: a simple export PYTHONPATH=. will do it. However, I don't like doing so, and don't like relative import. What are my options now?

The import mechanism is based on PYTHONPATH.
When you run python main.py, then the directory containing main.py is in PYTHONPATH, so all other packages there are importable as well.
When you run python formats/writer.py, then the formats directory is in PATHONPATH and its parent directory is not, so you cannot import modules and packages which are not in formats.
What you can do, is run writer module, but have the root directory in PATHONPATH and you can do that without even messing with environment variables:
cd /directory/in/which/main.py/is
python -m formats.writer
Unlike python formats/writer.py, which changes PYTHONPATH and runs writer.py, this keeps the default PYTHONPATH (current directory) and tells Python to look within that path for a module named formats.writer and run that as the main module.

Python - ModuleNotFoundError: No module named

I'm new in Python and I'm having the following error with this simple example:
This is my project structure:
python_project
.
├── lib
│   ├── __init__.py
│   └── my_custom_lib.py
└── src
├── __init__.py
└── main.py
And this is the error when I execute the src/main.py file:
☁ python_project python src/main.py
Traceback (most recent call last):
File "src/main.py", line 3, in <module>
from lib import my_custom_lib
ImportError: No module named lib
If I move the main.py file to the root and then I execute this file again, works... but is not working inside src/ directory
This is my main.py:
from lib import my_custom_lib
def do_something(message):
my_custom_lib.show(message)
do_something('Hello World!')
Note: When I execute the same code from Pycharm is working fine, but not from my terminal.

Your PYTHONPATH is set to the parent directory of the executed script. So if the executed script is inside a directory src, it will never be able to find the sister directory lib because it isn't within the path. There's a few choices;
Just move lib/ into src/ if it belongs to your code. If it's an external package, it should be pip installed.
Have a top-level script outside of src/ that imports and runs src.main. This will add the top-level directory to python path.
In src/main.py modify sys.path to include the top-level directory. This is usually frowned upon.
Invoke src/main.py as a module with python -m src.main which will add the top-level directory to the python path. Kind of annoying to type, plus you'll need to change all your imports.

If I may add to MarkM's answer, if you wanted to keep your current directory structure and still make it work, you could add a setup.py in your root dir, where you can use setuptools to create a package you could install.
If your file had something along the lines of:
# setup.py
from setuptools import find_packages, setup
setup(
name='foo',
version=`1.0.0`,
packages=find_packages(),
entrypoints={
'console_scripts': [
'foo=src.main:main',
],
},
)
And then you do pip install [--user] -e path/to/directory you'll get an "editable package" which will effectively a symlink to the package in your development directory, so any changes you make will not require a reinstall (unless of course you rejig package structure or add/remove/edit entry points).
This does assume your src/main.py has a main function.
You'll also need __init__.py files in your "package" directories, even in Python3, as otherwise Python assumes these are namespace packages (Won't go into detail) and the find_packages() call won't find them.
This will also allow your relative imports to work. Absolute imports will only work when invoking the script from your entry point but not when calling the script directly in your development directory.

You should have your main.py script above all python packages in your directory structure. Try to update your project to the following structure:
.
|__ main.py
|__ lib
| |__ __init__.py
| |__ your_custom_lib.py
|__ another_python_package
|__ __init__.py
|__ another_python_scripts
After that, python main.py in your project directory will work.

You are using the from a import b incorrectly. it should look like this:
import lib.my_custom_lib
The other method is used to import certain methods, functions, and classes from a module, not the module itself. To import a specific function from the my_custom_lib module it would look like this:
from lib.my_custom_lib import foo

Try using a relative import instead:
from ..lib import my_custom_lib

in my case in visual code
actual error in line 1
I didn't imported _typeshed module but by default it was their
so delete that module if you found in line 1

MarkM's answer is still excellent; I'm using PyPi now. I attempted to disambiguate what he was saying about "current directory". In case my confusion was an intended feature, here is how I did it.
vagrant#testrunner:~/pypath$ tree
.
├── proga
│   └── script1.py
└── progb
└── script1.py
script1.py is the same in both directories:
#!/usr/bin/env python3
import sys
print(sys.path)
Where I run it from makes no difference, PYTHONPATH prepends the directory containing the script I specify:
vagrant#testrunner:~/pypath/proga$ ./script1.py
['/home/vagrant/pypath/proga', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/vagrant/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']
vagrant#testrunner:~/pypath/proga$ ../progb/script1.py
['/home/vagrant/pypath/progb', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/vagrant/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']

For me importing with explicit paths works best in such situations. If you need to go up in the tree use '..'. The plumbing is a bit cumbersome, but it always works.
path = os.path.abspath(os.path.join(pathlib.Path(__file__).parent.absolute(), '..', 'subdir', 'myFile.py'))
loader = importlib.machinery.SourceFileLoader('myFile', path)
spec = importlib.util.spec_from_loader('myFile', loader)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# now use the module:
module.myMethod()
myClassInstance = module.myClass()

How to correctly make a python package?

This is my first time making a python package, and I am thoroughly confused about __init__.py, __main__.py, and their relation to making a package.
Here is my project structure:
package_name/
setup.py
README.md
LICENSE
package_name/
__init__.py
__main__.py
A.py
B.py
Class A in A.py depends on a class B in B.py.
Should I have both __init__.py and __main__.py?
What should be in the files?
What I have tried:
in A.py:
from B import B and from .B import B
The first allows me to run normally locally, but when I try to upload it to pypi and install it, I get ModuleNotFoundError: No module named 'B'
The second allows me to import it after installing it from pypi, but I can't run it normal locally.
My goal is to import Class A from the package with the following
from package_name import A
and be able to run my package locally.
Edit:
I am using Python 3.

Files named init.py are used to mark directories on disk as a Python package directories, you can let this empty most of the time. But let's say you have for example this file structure with the following code inside them:
Structure
package_name/
├── setup.py
├── package_name
│ ├── __init__.py
│ └── main.py
└── package_name.py
setup.py
#!/usr/bin/env python3
import sys
from setuptools import setup
setup(
name = "package_name",
version = "0.1",
packages=["package_name"],
package_data = {},
author="xxx",
author_email = "xxx#xxx.xx",
description = "The familiar example program in Python",
license = "BSD",
keywords= "example documentation tutorial",
)
package_name.py
#!/usr/bin/env python
import sys
import package_name.main
if __name__ == '__main__':
sys.exit(package_name.main.main())
main.py
#!/usr/bin/env python3
import sys
def main(argv=None):
if argv is None:
argv = sys.argv
print("Hello, world")
return 0
Go in the terminal to the package_name folder and type: "python3 package_name.py" to execute the code.
Output
Hello, world
The package.py goes to the main.py and execute the code that is in the main.py. If you want to import in your example try using for example "from package_name.A.py import [function name]" in the the python files you wish to access the functions. It should work.
I would like to hear from you if this helped you out and gave you a better understanding.

ModuleNotFound: No module named 'module_name'

My project structure is the following:
.
└── project name
├── project name
│ ├── __init__.py
│ ├── module.py
│
├── PACKAGE_A
│ ├── __init__.py
│ ├── PACKAGE_A.py
│ ├── module_a.py
│
In PACKAGE_A.py
from module_a import Some_Class
a = Some_Class()
class Another_Class:
# class code here
In module.py
"""
Notes
-----
https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
"""
# Standard library imports
import os, sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
# Local application imports
from PACKAGE_A.PACKAGE_A import Another_Class
from PACKAGE_A.module_a import some_function
While module_a.py and PACKAGE_A.py run without problem, running module.py fails with:
Traceback (most recent call last):
File "path\to\project name\project name\module.py", line 12, in <module>
from PACKAGE_A.PACKAGE_A import Another_Class
File "path\to\project name\PACKAGE_A\PACKAGE_A.py", line 1, in <module>
from module_a import Some_Class
ModuleNotFoundError: No module named 'module_a'
What am I doing wrong here?

You need to change your import statement in PACKAGE_A.py from:
from module_a import Some_Class
to:
from PACKAGE_A.module_a import Some_Class
The reason is that you are adding the path\to\project name\ to sys.path, but you have no module_a.py in path\to\project name\, and path\to\project name\PACKAGE_A (where module_a.py resides) is not in sys.path.
As for why you succeed in running everything in PACKAGE_A, it's because Python adds the directory containing the script you are running to the list (as explained by gaFF).
I would recommend you read a bit more about python imports, if the doc seems too cluttered, you can check this link.
This is a personal preference, but I find it simpler to add the root directory of the project to the PYTHONPATH environment variable and then running all the scripts from that directory's level and changing the import statements accordingly. In your example, the root directory would be path\to\project name\.

import search for your packages in specific places, listed in sys.path. See the doc for the full details.
The current directory is always appended to this list, that's why you succeed to run everything inside PACKAGE_A. But from project name, there is no way to know where to find PACKAGE_A.
Solutions include:
use relative import
always run from the root directory (and start all your imports from the root directory)
add the root directory to the environment variable PYTHONPATH (same)
use a tool which sets PYTHONPATH when you enter your virtual environment (same)
...
and depends on your project and your needs.

Python3 imports

I am looking for a way to import modules so that I can run a script both from the sub-folder project/v0 and from the root folder project
My file structure in python 3.6 (which is why there are no init files)
project
├── api.py
├── v0
│   ├── SearchEngine.py => contains SearchEngine class
│   └── SearchEngineBE.py
My SearchEngineBE.py module contains
from SearchEngine import SearchEngine
My api.py module contains
from v0.SearchEngineBE import SearchEngineBE
step1: When from project/v0 I run python3 SearcheEngineBE.py my module is correctly imported and everything goes well.
step2: However, when from project I run python3 run api.py I get the error:
Traceback (most recent call last):
File "api.py", line 3, in <module>
from v0.SearchEngineBE import SearchEngineBE
File "/xxx/project/v0/SearchEngineBE.py", line 3, in <module>
from SearchEngine import SearchEngine
ModuleNotFoundError: No module named 'SearchEngine'
How can I fix this so that both step 1 and step 2 would work ?

Non-relative imports are searched by the interpreter in the current directory (and any additional search paths).
You could use relative imports in your SearchEngineBE.py file to let the interpreter know you want the relative module, and not a module off the import path:
# SearchEngineBE.py
from .SearchEngine import SearchEngine
The . lets the interpreter know that you are referencing a module relative to the current module. You'll need at least a blank __init__.py file in the same directory as SearchEngine.py for relative imports to work though:
The __init__.py files are required to make Python treat directories
containing the file as packages.
https://docs.python.org/3/tutorial/modules.html
See this guide for some more discussion: https://chrisyeh96.github.io/2017/08/08/definitive-guide-python-imports.html#absolute-vs-relative-import
Edit:
Without a root package, this won't work. See this post for an alternative approach: Importing modules from a neighbouring folder in Python

Add __init__.py to make it a package.
(it could be empty file : __init__.py)
project
├── __init__.py
├── api.py
├── v0
│ ├── SearchEngine.py => contains SearchEngine class
│ └── SearchEngineBE.py
Edit 1:
try :
from project.v0.SearchEngine import SearchEngine