Develop Reference

Python Import function from another file not working

I'm facing a problem with importing functions from another file that is not working.
Here is the simplify tree of my folders, located on /var/www/html/opencaptureforinvoices/ :
├── custom
│   └── test
│   └── src
│   └── backend
│   └── process_queue.py
└── src
└── backend
└── main.py
I run the process_queue.py script with the following command, using Kuyruk (lib to enqueue process)
cd /var/www/html/opencaptureforinvoices/custom/test || exit
/usr/local/bin/kuyruk --app src.backend.process_queue_verifier.kuyruk worker --queue
The problem is that I need function from main.py. I use import like this :
from src.backend.main import create_classes_from_custom_id, check_file, timer, str2bool
Before posting I tried to rewrite the path to the root of custom & src using sys.path.append or sys.path.insert or os.path.chdir but none of them working, the application said to me :
ModuleNotFoundError: No module named 'src.backend.main'
Here is the command I tried to move to root folder :
os.chdir('/var/www/html/opencaptureforinvoices/')
sys.path.append('/var/www/html/opencaptureforinvoices/')
sys.path.insert(0, '/var/www/html/opencaptureforinvoices/')
Any ideas ?
Thanks

Have you set up your subdirectories as packages using the __init__.py file?
https://docs.python.org/3/tutorial/modules.html#packages
The __init__.py files are required to make Python treat directories
containing the file as packages. This prevents directories with a
common name, such as string, unintentionally hiding valid modules that
occur later on the module search path. In the simplest case,
__init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable,
described later.
Users of the package can import individual modules from the package,
for example:
import sound.effects.echo

First solution:
Make sure the folder also contains an __ init __.py, this allows it to be included as a package.
Second:
When importing a file, Python only searches the directory that the entry-point script is running from and sys.path which includes locations such as the package installation directory.
But you can add to the Python path at runtime:
some_file.py
import sys
sys.path.append('/path/to/application/app/folder')
import file

ValueError: attempted relative import beyond top-level package python

I have a directory structure like this.
Chatbot/
utils/
abc.py
projects/
proj1/
utils/
__init__.py
data_process.py
components/
class1.py
I have two utils folders in my structure, one at the top level and one inside my projects folder.
Now I want to import data_process.py file inside class1.py. So I tried like this
from utils.data_process import DataProcess
but it is referencing the top-level utils folder and even VSCode is not recognizing it. I tried creating __init__.py file inside the utils folder but still did not work.
I tried with empty __init__.py and then placing this content
from . import data_process
__all__ = ['data_proces']
then
from .data_process import DataPreprocess
__all__ = ['DataPreprocess']
then I tried
from ..utils.data_process import DataProcess
VSCode is recognizing this but it is not working and throws the error
ValueError: attempted relative import beyond top-level package
Even I tried changing the name utils to some other name but still the same issue
How can I solve this?

A python module is defined by a special file called __init__.py and this file should be present in all the subdirectories. So, your file structure would be:
Chatbot/
__init__.py
utils/
__init__.py
abc.py
projects/
__init__.py
proj1/
__init__.py
utils/
__init__.py
data_process.py
components/
__init__.py
class1.py
class2.py
Now, you can do a relative import like:
Use . for importing something within the same directory. Example:
# file Chatbot/projects/proj1/components/class2.py
from .class1 import *
Similarly, use .. for two-level, and ... for three-level, and so on!
For instance:
# file Chatbot/projects/proj1/components/class2.py
from ..utils.data_process import * # import from data_process.py
# there will be four "." earlier I made a counting mistake
from ....utils.abc import * # import something from abc.py
Coding Convention
When writing a python module/package, you might want to follow PEP8 Convention.
Moreover, I believe you are trying to do different projects using your package Chatbot, is that correct? Then, it is a good practice that you set PYTHONPATH for Chatbot and do all the projects, and imports seperately.
EDIT: Uploading Dummy File
I'm able to work on this project seamlessly even with using two utils directories. Project Structure:
main file and its output:
# file main.py
from chatbot.utils.abc import hello as a1
from chatbot.projects.proj1.components.class1 import super_hello
from chatbot.projects.proj1.utils.data_process import hello as b1
print(a1(), b1())
print(super_hello())
Similarly, if you have chatbot under PYTHONPATH you can call the project from anywhere from your device.
Code details and files are added into my github account for details.

sys.path is where Python searches to find modules and packages and It does it in order.
So you can put ...../proj1/ at the beginning of the list, when python start searching it will find the utils folder in that path first !
import sys
sys.path.insert(0, r'...../proj1/')
But this cause another problem, python always find utils in that folder first, that's not what you want.
solution number 1
Absolute import:
from Chatbot.projects.proj1.utils.data_process import DataProcess
solution number 2
use relative imports which is mentioned by #Mr.Hobo.

How to correctly do relative imports with nested packages?

Yes I read the answer to this Relative imports for the billionth time. No I did not find it really useful as I tried both solutions and it did not work for me. See below.
I have the below folder structure.
(I do not have any __init__.py, as I understood they're not required anymore. I have Python 3.7)
project
├── subpackage1
└── subpackage2
├── launch
| └── launchFunc.py
| └── launchFunc.sh
└── src
└── somemodule.py
launchFunc.py:
from somemodule import func
func()
launchFunc.sh:
#!/bin/bash
python -m launch.launchFunc
Then I open command line and I cd into project/subpackage2/ and I run the below command:
./launch/launchFunc.sh
and I get the below error:
from somemodule import func
ModuleNotFoundError: No module named 'somemodule'
If I replace from somemodule import func with from ..src.somemodule import func, I get:
ValueError: attempted relative import beyond top-level package
What is the correct way to import and/or run scripts in python ?
Isn't there a way to have all functions automatically imported to all required modules ?

Ok I have found out that by:
always using the full path to import: import func from subpackage2.somemodule
always running things from the project folder
it will work fine.
I think this is a rule I should adopt for all my projects.

ImportError : Attempted relative import with no known parent package [duplicate]

This question already has answers here:
Relative imports in Python 3
(31 answers)
Closed 4 months ago.
I am learning to program with python and I am having issues with importing from a module in a package. I am usingvisual studio code with Python 3.8.2 64 bit.
My Project Directory
.vscode
├── ecommerce
│   ├── __init__.py
│ ├── database.py
│   ├── products.py
│   └── payments
│ ├── __init__.py
│   ├── authorizenet.py
│   └── paypal.py
├── __init__.py
└── main.py
in the ecommerce/products.py file I have:
#products.py
from .database import Database
p = Database(3,2)
So that I can import the Database class from the ecommerce/database.py file. But I get error
ImportError : Attempted relative import with no known parent package

It seems, from Python docs and experimenting, that relative imports (involving ., .. etc) only work if
the importing module has a __name__ other than __main__, and further,
the __name__ of the importing module is pkg.module_name, i.e., it has to be imported from above in the directory hierarchy (to have a parent pkg as part of it's __name__.)
OR
the importing module is being specified via module syntax that includes a parent pkg as python -m pkg.module, in which case it's __name__ is still __main__, so it is being run as a script, yet relative imports will work. Here __package__ is set and used to find the parent package while __name__ is __main__; more here.
[After all that, it appears that __package__ and sys.path are key to determining if/how relative imports work. __name__ indicates script or module(i.e., __main__ or module_name). __package__ indicates where in the package the relative imports occur with respect to, and the top of __package__ needs to be in sys.path.]
So, continuing with #AmitTendulkar 's example, if you run this as > python main.py or > python -m main or > python -m ecommerce.products from the project root directory, or enter interactive python from that root directory and import main, or import ecommerce.products the relative imports in products.py will work.
But if you > python products.py or > python -m products from within ecommerce directory, or enter interactive python from that ecommerce directory and import products they will fail.
It is helpful to add
print("In module products __package__, __name__ ==", __package__, __name__)
etc. in each file to debug.
UPDATE:
How imports work depend on sys.path and __package__, not on __name__.
Issued from /home/jj, > python sub/mod.py has a sys.path, __package__ of /home/jj/sub, None -absolute imports of modules in sys.path work, relative imports fail.
> python -m sub.mod has sys.path, __package__ of /home/jj, sub -absolute imports of modules in sys.path work, relative imports work relative to sys.path + __package__.
It is more helpful to add
import sys
print("In module products sys.path[0], __package__ ==", sys.path[0], __package__)
etc. in each file to debug.

Since you are using Python 3.8 version, the imports work a little differently, but I think this should work:
Use either:
from database import Database
#Database is the class
or try:
import database.Database
lastly, this one is very secure and best practice possibly:
from . import Database
# The '.' (dot) means from within the same directory as this __init__.py module grab the Database class.

i had a similar issue on Windows, and what helped me was (adapted to your directory):
# local imports
import sys
sys.path.append(r"C:\path\to\your\project")
from ecommerce.database import Database

Considering the below basic file structure
├── ecommerce
│ ├── __init__.py
│ ├── database.py
| └── products.py
└── main.py
I am assuming you are running python main.py from the project root.
Here is the text copied from the Python tutorial that explains the basic rule around relative import,
Note that relative imports are based on the name of the current module. Since the name of the main module is always __main__, modules intended for use as the main module of a Python application must always use absolute imports.
So the below code will work,
# main.py
import ecommerce.products
# or to use it directly
# from ecommerce.products import my_product
ecommerce.products.my_product()
Your product.py might look like,
# ecommerce/products.py
from .database import Database
def my_product():
p = Database(3, 2)
And database.py will look like below,
# ecommerce/database.py
class Database():
def __init__(self, x, y):
self._x = x
self._y = y
print('Inside DB init')
# Rest of the methods...
You will now get,
> python main.py
Inside DB init
Ideally the __init__.py file at root is not required as the package name is starting from ecommerce.
You can also run python -m ecommerce.products command to directly call products.py. But that won't yield any output as we are not calling the my_product() function (only defining it).
Calling python ecommerce/products.py will not work as the name of the current module will then become __main__ and not ecommerce. The relative importing only works when used within the current package (so in your main script you always need to import your ecommerce package).

To allow the use of relative imports, you need to "turn your code into a package". This means all of 1) putting an __init__.py in the top directory of your code (in your example .vscode) and 2) adding the full (absolute) path to the parent directory of the top directory (i.e., the parent of your directory .vscode) to your PYTHONPATH and 3) setting the __package__ variable in your Python program to the name of the directory that contains __init__.py, in your case ".vscode".
You should then be able to use in ecommerce/products.py:
from .ecommerce.database import Database
I'm not sure why the dot is in the name .vscode - is that part of the directory name, or part of the lines in your directory tree? If the latter, replace .vscode with vscode above.

Try this...
from ecommerce.database import Database

This may seem like a hack but it actually work, this happens because of bad package structure
In you case try importing
from .vs_code.ecommerce import database
Now wherever you want to call the class constructor do this:
database.Database()
you can assign this to a variable and use it.

Presumably, you make executable the file "products.py", which violates the very concept of packages, the package ceases to be a package and relative import does not work. You must call this outside the package.
Here I am also not sure whether the root directory name can start with a dot like ".vscode".

Relative import in Python 3 is not working [duplicate]

This question already has answers here:
Python3 correct way to import relative or absolute?
(2 answers)
Closed 2 years ago.
I have the following directory:
mydirectory
├── __init__.py
├── file1.py
└── file2.py
I have a function f defined in file1.py.
If, in file2.py, I do
from .file1 import f
I get the following error:
SystemError: Parent module '' not loaded, cannot perform relative
import
Why? And how to make it work?

Launching modules inside a package as executables is a bad practice.
When you develop something you either build a library, which is intended to be imported by other programs and thus it doesn't make much sense to allow executing its submodules directly, or you build an executable in which case there's no reason to make it part of a package.
This is why in setup.py you distinguish between packages and scripts. The packages will go under site-packages while the scripts will be installed under /usr/bin (or similar location depending on the OS).
My recommendation is thus to use the following layout:
/
├── mydirectory
| ├── __init__.py
| ├── file1.py
└── file2.py
Where file2.py imports file1.py as any other code that wants to use the library mydirectory, with an absolute import:
from mydirectory.file1 import f
When you write a setup.py script for the project you simply list mydirectory as a package and file2.py as a script and everything will work. No need to fiddle with sys.path.
If you ever, for some reason, really want to actually run a submodule of a package, the proper way to do it is to use the -m switch:
python -m mydirectory.file1
This loads the whole package and then executes the module as a script, allowing the relative import to succeed.
I'd personally avoid doing this. Also because a lot of people don't even know you can do this and will end up getting the same error as you and think that the package is broken.
Regarding the currently accepted answer, which says that you should just use an implicit relative import from file1 import f because it will work since they are in the same directory:
This is wrong!
It will not work in python3 where implicit relative imports are disallowed and will surely break if you happen to have installed a file1 module (since it will be imported instead of your module!).
Even if it works the file1 will not be seen as part of the mydirectory package. This can matter.
For example if file1 uses pickle, the name of the package is important for proper loading/unloading of data.

When launching a python source file, it is forbidden to import another file, that is in the current package, using relative import.
In documentation it is said:
Note that relative imports are based on the name of the current module. Since the name of the main module is always "__main__", modules intended for use as the main module of a Python application must always use absolute imports.
So, as #mrKelley said, you need to use absolute import in such situation.

since file1 and file2 are in the same directory, you don't even need to have an __init__.py file. If you're going to be scaling up, then leave it there.
To import something in a file in the same directory, just do like this
from file1 import f
i.e., you don't need to do the relative path .file1 because they are in the same directory.
If your main function, script, or whatever, that will be running the whole application is in another directory, then you will have to make everything relative to wherever that is being executed.

myproject/
mypackage
├── __init__.py
├── file1.py
├── file2.py
└── file3.py
mymainscript.py
Example to import from one file to another
#file1.py
from myproject import file2
from myproject.file3 import MyClass
Import the package example to the mainscript
#mymainscript.py
import mypackage
https://docs.python.org/3/tutorial/modules.html#packages
https://docs.python.org/3/reference/import.html#regular-packages
https://docs.python.org/3/reference/simple_stmts.html#the-import-statement
https://docs.python.org/3/glossary.html#term-import-path
The variable sys.path is a list of strings that determines the interpreter’s search path for modules. It is initialized to a default path taken from the environment variable PYTHONPATH, or from a built-in default if PYTHONPATH is not set. You can modify it using standard list operations:
import sys
sys.path.append('/ufs/guido/lib/python')
sys.path.insert(0, '/ufs/guido/myhaxxlib/python')
Inserting it at the beginning has the benefit of guaranteeing that the path is searched before others (even built-in ones) in the case of naming conflicts.