Develop Reference

ValueError: attempted relative import beyond top-level package python

I have a directory structure like this.
Chatbot/
utils/
abc.py
projects/
proj1/
utils/
__init__.py
data_process.py
components/
class1.py
I have two utils folders in my structure, one at the top level and one inside my projects folder.
Now I want to import data_process.py file inside class1.py. So I tried like this
from utils.data_process import DataProcess
but it is referencing the top-level utils folder and even VSCode is not recognizing it. I tried creating __init__.py file inside the utils folder but still did not work.
I tried with empty __init__.py and then placing this content
from . import data_process
__all__ = ['data_proces']
then
from .data_process import DataPreprocess
__all__ = ['DataPreprocess']
then I tried
from ..utils.data_process import DataProcess
VSCode is recognizing this but it is not working and throws the error
ValueError: attempted relative import beyond top-level package
Even I tried changing the name utils to some other name but still the same issue
How can I solve this?

A python module is defined by a special file called __init__.py and this file should be present in all the subdirectories. So, your file structure would be:
Chatbot/
__init__.py
utils/
__init__.py
abc.py
projects/
__init__.py
proj1/
__init__.py
utils/
__init__.py
data_process.py
components/
__init__.py
class1.py
class2.py
Now, you can do a relative import like:
Use . for importing something within the same directory. Example:
# file Chatbot/projects/proj1/components/class2.py
from .class1 import *
Similarly, use .. for two-level, and ... for three-level, and so on!
For instance:
# file Chatbot/projects/proj1/components/class2.py
from ..utils.data_process import * # import from data_process.py
# there will be four "." earlier I made a counting mistake
from ....utils.abc import * # import something from abc.py
Coding Convention
When writing a python module/package, you might want to follow PEP8 Convention.
Moreover, I believe you are trying to do different projects using your package Chatbot, is that correct? Then, it is a good practice that you set PYTHONPATH for Chatbot and do all the projects, and imports seperately.
EDIT: Uploading Dummy File
I'm able to work on this project seamlessly even with using two utils directories. Project Structure:
main file and its output:
# file main.py
from chatbot.utils.abc import hello as a1
from chatbot.projects.proj1.components.class1 import super_hello
from chatbot.projects.proj1.utils.data_process import hello as b1
print(a1(), b1())
print(super_hello())
Similarly, if you have chatbot under PYTHONPATH you can call the project from anywhere from your device.
Code details and files are added into my github account for details.

sys.path is where Python searches to find modules and packages and It does it in order.
So you can put ...../proj1/ at the beginning of the list, when python start searching it will find the utils folder in that path first !
import sys
sys.path.insert(0, r'...../proj1/')
But this cause another problem, python always find utils in that folder first, that's not what you want.
solution number 1
Absolute import:
from Chatbot.projects.proj1.utils.data_process import DataProcess
solution number 2
use relative imports which is mentioned by #Mr.Hobo.

Importing from parent directory python

I have the following structure:
app/
code/
script.py -- has a function called func
main.py
How can I import script.py from main.py ?
I tried from code.script import func and I got ModuleNotFoundError: No module named 'code.script'; 'code' is not a package

Place a __init__.py file in the code directory. This will allow your main.py code to import it as a module like you have tried there.

Indeed the best way is to add __init__.py in code directory because when a regular package is imported __init__.py file is implicitly executed and the objects it defines are bound to names in the package’s namespace.
FYI, as an alternative you can also to this in your main.py before the import:
import sys
sys.path.append("/path/to/script.py")

Why do I have to use a relative import in a package's __init__.py?

Setup
test/
main.py
pkg/
a.py
__init__.py
main.py contains:
import pkg
pkg.a
__init__.py contains:
from . import a
main.py can be run without errors.
Question
Changing the content of __init__.py to
import a
gives the following error when running main.py:
Traceback (most recent call last):
File "C:/Users/me/PycharmProjects/test/main.py", line 1, in <module>
import pkg
File "C:\Users\me\PycharmProjects\test\pkg\__init__.py", line 1, in <module>
import a
ModuleNotFoundError: No module named 'a'
Interestingly, __init__.py can be executed directly with python __init__.py without errors.
What's going on?

When you run a python script, it's parent folder is added to sys.path
run main.py: sys.path[0] = '../test'
run init.py: sys.path[0] = '../test/pkg'
Your case: You try to "absolute-like" import a in __init__.py but the parent folder of a.py - which is '../test/pkg' - is not in the sys.path when you run main.py. This is why you get an error. However, your absolute import is incomplete as it should always start at the top level folder, e.g.
from test.pkg import a
Final answer to your question: You don't have to use relative imports!
See: PEP-8: Absolute imports are recommended, as they are usually more readable and tend to be better behaved (or at least give better error messages) if the import system is incorrectly configured (such as when a directory inside a package ends up on sys.path).
And keep in mind that relative imports don't work in a top-level-script when __name__ = "__main__", but from imported modules only.
You can learn more about absolute and relative imports here:
Absolute vs. explicit relative import of Python module
https://realpython.com/absolute-vs-relative-python-imports/

I suppose you are using Pycharm? Then that's one of the confusion cause.
For example, let's say your directory looks like this
project1
p1.py
test/
__init__.py
main.py
pkg/
a.py
__init__.py
If you run (F10) the main.py your default working directory will be project1/test, which does not contain the a.py so import a will not find anything.
But if you run (F10) the pkg/__init__.py your working directory will be project1/test/pkg which has the a.py, and it works like what you tested.
So in these situation, if you use from . import a it will look for the directory that file is, project1/test/pkg in this case, which will always work regardless your working directory.

Best way to import a package located in a sibling directory

My file stucture is :
top\
my_package\
__init__.py
functions.py
scripts\
test.py
main.py
I would like to import the content of functions.py in test.py.
In main.py, I can import functions.py with from my_package.functions import ....
I was expecting to be able to import functions.py in test.py with something like from ..my_package.functions import ... but it raises the following error :
SystemError: Parent module '' not loaded, cannot perform relative import
top directory shoudln't be a package because I want to be able to run main.py without being running a script in a package.
What is the proper/pythonic way import functions.py from test.py ?
I could add my_package to the PYTHONPATH, but my code would be less portable. I'm using Python 3.5

This is happening because as far as python is concerned my_package and scripts directories are not related to each other.
To solve your problem you can add __init__.py to your top directory and try using
from ..function import *

How do I correctly import packages with py.test?

I have the following layout:
/spamalot
/spam
__init__.py
spam.py
/spam_on_eggs
__init__.py
spam_on_eggs.py
/tests
test_spam.py
Spam just so happens to be a flask application.
Within spam.py I have
import spam_on_eggs.spam_on_eggs as eggs
# Other Flask setup & application code here.
And this works fine - from the spamalot directory I'm able to run python spam/spam.py
However, when I start to throw tests into the mix, it's not as awesome.
In my test_spam.py file I have:
import spam.spam
test_client = spam.spam.app.test_client()
def test_it_fails():
assert False
However, rather than failing where I would expect, it fails on the import line:
/spamalot/ $ py.test
# some output
E ImportError
I can fix this by putting __init__.py in my /tests folder, but then I get a different ImportError:
spam/spam.py:1: in <module>
> import spam_on_eggs.spam_on_eggs as eggs
E ImportError: No module named 'spam_on_eggs'
I can solve that one by changing the line to:
from spam.spam_on_eggs import spam_on_eggs
Which allows me to test but then I break my ability to run $ python spam/spam.py - because I get
ImportError: no module named 'spam'
Obviously I have a gap in my understanding of how module imports work and how py.test works with this system.
What am I missing?
Is it even possible to have the layout I've described - and be able to run both py.test and my server from the spamalot directory?

py.test will always use the shortest directory path with an __init__.py in it.
Put a __init__.py into spamalot and you can import spamalot.spam.spam.
Reference: choosing a test layout