This question already has answers here:
Python3 correct way to import relative or absolute?
(2 answers)
Closed 2 years ago.
I have the following directory:
mydirectory
├── __init__.py
├── file1.py
└── file2.py
I have a function f defined in file1.py.
If, in file2.py, I do
from .file1 import f
I get the following error:
SystemError: Parent module '' not loaded, cannot perform relative
import
Why? And how to make it work?
Launching modules inside a package as executables is a bad practice.
When you develop something you either build a library, which is intended to be imported by other programs and thus it doesn't make much sense to allow executing its submodules directly, or you build an executable in which case there's no reason to make it part of a package.
This is why in setup.py you distinguish between packages and scripts. The packages will go under site-packages while the scripts will be installed under /usr/bin (or similar location depending on the OS).
My recommendation is thus to use the following layout:
/
├── mydirectory
| ├── __init__.py
| ├── file1.py
└── file2.py
Where file2.py imports file1.py as any other code that wants to use the library mydirectory, with an absolute import:
from mydirectory.file1 import f
When you write a setup.py script for the project you simply list mydirectory as a package and file2.py as a script and everything will work. No need to fiddle with sys.path.
If you ever, for some reason, really want to actually run a submodule of a package, the proper way to do it is to use the -m switch:
python -m mydirectory.file1
This loads the whole package and then executes the module as a script, allowing the relative import to succeed.
I'd personally avoid doing this. Also because a lot of people don't even know you can do this and will end up getting the same error as you and think that the package is broken.
Regarding the currently accepted answer, which says that you should just use an implicit relative import from file1 import f because it will work since they are in the same directory:
This is wrong!
It will not work in python3 where implicit relative imports are disallowed and will surely break if you happen to have installed a file1 module (since it will be imported instead of your module!).
Even if it works the file1 will not be seen as part of the mydirectory package. This can matter.
For example if file1 uses pickle, the name of the package is important for proper loading/unloading of data.
When launching a python source file, it is forbidden to import another file, that is in the current package, using relative import.
In documentation it is said:
Note that relative imports are based on the name of the current module. Since the name of the main module is always "__main__", modules intended for use as the main module of a Python application must always use absolute imports.
So, as #mrKelley said, you need to use absolute import in such situation.
since file1 and file2 are in the same directory, you don't even need to have an __init__.py file. If you're going to be scaling up, then leave it there.
To import something in a file in the same directory, just do like this
from file1 import f
i.e., you don't need to do the relative path .file1 because they are in the same directory.
If your main function, script, or whatever, that will be running the whole application is in another directory, then you will have to make everything relative to wherever that is being executed.
myproject/
mypackage
├── __init__.py
├── file1.py
├── file2.py
└── file3.py
mymainscript.py
Example to import from one file to another
#file1.py
from myproject import file2
from myproject.file3 import MyClass
Import the package example to the mainscript
#mymainscript.py
import mypackage
https://docs.python.org/3/tutorial/modules.html#packages
https://docs.python.org/3/reference/import.html#regular-packages
https://docs.python.org/3/reference/simple_stmts.html#the-import-statement
https://docs.python.org/3/glossary.html#term-import-path
The variable sys.path is a list of strings that determines the interpreter’s search path for modules. It is initialized to a default path taken from the environment variable PYTHONPATH, or from a built-in default if PYTHONPATH is not set. You can modify it using standard list operations:
import sys
sys.path.append('/ufs/guido/lib/python')
sys.path.insert(0, '/ufs/guido/myhaxxlib/python')
Inserting it at the beginning has the benefit of guaranteeing that the path is searched before others (even built-in ones) in the case of naming conflicts.
Related
Imagine a package with this structure:
└── main_package
├── __init__.py
├── subpackage1
│ ├── __init__.py
│ └── module1.py
└── subpackage2
├── __init__.py
└── module2.py
What is the best way of importing a function in module1 to module2?
Should the __init__.py files contain something?
The best solution I have found is to include in module2
import sys
sys.path.append('./')
from main_package.subpackage1.module1 import fun1
And when running module2 from the directory in which main_package is it works.
But I feel like this does not take advantage of the fact that they are packages. And also it would be nice to be able to run this from wherever I want if possible.
The most reliable is to always use absolute imports. In this case for module 1:
from MainPackage.module2 import some_function
In module1_1 that would be:
from MainPackage.Subpackage2.module2_1 import some_function
Note that the package naming convention is to have all lower case letters and in between words, an underscore (_) can be used. In your specific example, MainPackage would become main_package and so on.
For your first question:
What is the best way of importing a function in module1 to module2?
You may either use relative or absolute imports. Personally, I prefer to use relative imports only within the context of a package I'm working on - to me this highlights when internal packages are related. In addition (and this is environment specific), certain environments will not recognize main_package until you install the package (i.e. with pip install -e . during development).
Absolute import in module2: from main_package.subpackage1.module1 import my_function
Relative import in module2: from ..subpackage1.module1 import my_function
For your second question:
Should the init.py files contain something?
They can, and that may provide you with shortcuts for imports. It is common practice to expose general and/or first-class functions/classes on the package level, which essentially is just importing them in __init__.py.
For example, in subpackage1\__init__.py you could:
from .module1 import my_function
And then in subpackage2\module2.py, you could simply use from ..subpackage1 import my_function (relative import) or from main_package.subpackage1 import my_function.
You may also be interested in the topic from the original Python documentation - import/submodules, Importing * From a Package (for additional reading on the import system, followed by intra-package imports), PEP328 (the rationale for relative imports) and Guido's thoughts regardings relative imports.
In Maple importing a module in the main file of your module will automatically share it with the other files and submodules of your module. You can also call the functions that you defined in different files of the same module as far as you have linked them by just including the address to the files of the module in the main file. So you only type the addresses in one file and you do not need to write them in other files of your module. In my opinion that makes life very easier.
In Python I couldn't do the same as in Maple and here is what I do in my current practice. Let me explain on a toy example. There is a folder containing a python file main.py and the package structure as you showed in your question. The content of each file is as follows.
main.py
import main_package
main_package.fun2()
The __init__.py of main_package.
from .subpackage1.module1 import fun1
from .subpackage2.module2 import fun2
The two init files of subpackages are empty.
module1.py
import sympy
x = sympy.Symbol('x')
def fun1(polynomial):
return(sympy.degree(polynomial, gen = x))
module2.py
import sympy
from ..subpackage1.module1 import fun1
def fun2():
polynomial = sympy.sympify(input("Type a univariate polynomial with the variable x and integer coefficients:"))
print(f"Degree of your polynomial in x is {fun1(polynomial)}.")
Now, instead of running the __init__.py of main_package, I run main.py in my VSCode. Note that my main.py file is next to main_package folder, not inside of it.
Again, I think it is sad that I have to write import sympy in each of the two files module1.py and module2.py.
I am very interested if someone someday shows a way so that one can write import sympy and x = sympy.Symbol('x') only in the top parent __init__.py and avoid writing it in the two files module1.py and module2.py.
I'm trying to import a class in a different directory to another file, but can't seem to get it to work. I know this question has been asked a lot and I have looked through multiple stackoverflow solutions and at https://docs.python.org/3/tutorial/modules.html#packages
1: Importing files from different folder
2: import python file in another directory failed
I want to try to just use the method containing just __init__.py file instead of doing an import sys
My directory structure is as follows:
django_vue/
__init__.py
devices/
__init__.py
models.py
lib/
__init__.py
my_file.py
I'm trying to import the class Device from /django_vue/devices/models.py to /django_vue/lib/my_file.py by:
from devices.models import Device
However when I do that I still get the error:
from devices.models import Device
ModuleNotFoundError: No module named 'devices'
I'm not sure what I'm dong wrong since I already have the __init__ file in both directories. Any help is appreciated. Also I'm running python 3.6.
This is the folder structure I'm working with.
.
└── django_vue
├── devices
│ └── models.py
└── lib
└── file.py
When you run
$ python file.py
python has no way of knowing what's outside the directory.
python can't go back and then into devices/ just like that.
The easiest way to solve this would be to add the folder devices/ to sys.path. When python imports a module, it searches for the module from sys.path. Adding the path to devices/ would make it available for imports.
Here are my files.
# models.py
Device = 'device'
# file.py
import sys
sys.path.append('..') # adds the parent dir (which is django-vue/) to path
# django-vue dir has devices/ so now this is available for imports
# importing this works now
from devices.models import Device
print(Device)
Output
django_vue/lib$ python3 file.py
device
Think about it your are inside my_file.py and import something called devices.
How can python know where the name devices has come from.
It won't search your entire Drive for that module/package
Relative Import
use a relative import instead. write from ..devices.models import Device. This is telling python to go up one directory to the parent directory and that's where it will find the devices package. Your lib module should now work as a module
If you however run the my_file.py package directly (as in python C:/django_vue/lib/my_file.py)
You will still get an error. Not the same error; the new error will be something like
ImportError: attempted relative import with no known parent package
This is happening because you are actually running my_file.py
If you think about it why would you want to run my_file.py by itself when it is clearly a part of a package. Maybe you are just testing to see if the import works when you use your package. The problem with this is that it makes it seem like your packages relative imports don't work even though this actually works.
Create a main.py in django_vue and write from lib import my_file. This will run your my_file.py and you will notice there is no error.
What's happening here
Have you heard of __package__?
if you put print(str(__package__)) in your my_file.py and run my_file.py directly you will see that it prints None.
However if you run main.py (that you just created) you will see that when It reaches my_file.py, __package__ will actually be defined to something.
Ahhh... you see now it all makes sense; The error you originally got said something about no known parent package. If __package__ is undefined that means there is no relative parent package because the file was obviously run directly instead of as part of a package.
Consider Absolute imports
you also might want to consider using absolute imports because if you are working on the package you might change it directory structure while developing. so you need to keep changing the import references on the affected files.
Although you can find IDE's with python extensions that automatically to this as you change your directory. I believe VS Code does this automatically.
Replace the __init__ files with __main__.
I'm trying to keep a data science project well-organized so I've created a directory inside my src directory called utils that contains a file called helpers.py, which contains some helper functions that will be used in many scripts. What is the best practice for how I should import func_name from src/utils/helpers.py into a file in a totally different directory, such as src/processing/clean_data.py?
I see answers to this question, and I've implemented a solution that works, but this feels ugly:
sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))))
Am I doing this right? Do I need to add this to every script that wants to import func_name, like train_model.py?
My current project folder structure:
myproject
/notebooks
notebook.ipynb
/src
/processing
clean_data.py
/utils
helpers.py
/models
train_model.py
__init__.py
Example files:
# clean_data.py
import os
import sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))))
from src.utils.helpers import func_name
func_name()
# helpers.py
def func_name():
print('I'm a helper function.')
The correct way to do it is to use __init__.py, setup.py and the setuptools Python package:
myPackage/
myPackage/
__init__.py
setup.py
This link has all the steps.
First of all, let me describe you the differences between a Python module & a Python package so that both of us are on the same page. ✌
A module is a single .py file (or files) that are imported under one import and used. ✔
import aModuleName
# Here 'aModuleName' is just a regular .py file.
Whereas, a package is a collection of modules in directories that give a package hierarchy. A package contains a distinct __init__.py file. ✔
from aPackageName import aModuleName
# Here 'aPackageName` is a folder with a `__init__.py` file
# and 'aModuleName', which is just a regular .py file.
Therefore, when we have a project directory named proj-dir of the following structure ⤵
proj-dir
--|--__init__.py
--package1
--|--__init__.py
--|--module1.py
--package2
--|--__init__.py
--|--module2.py
🔎 Notice that I've also added an empty __init__.py into the proj-dir itself which makes it a package too.
👍 Now, if you want to import any python object from module2 of package2 into module1 of package1, then the import statement in the file module1.py would be
from package2.module2 import object2
# if you were to import the entire module2 then,
from package2 import module2
I hope this simple explanation clarifies your doubts on Python imports' mechanism and solves the problem. If not then do comment here. 😊
First of all let me clarify you that importing an entire module, if you are going to use a part of it, then is not a good idea. Instead of that you can use from to import specific function under a library/package. By doing this, you make your program efficient in terms of memory and performance.
To know more refer these:
'import module' or 'from module import'
difference between import and from
Net let us look into the solution.
Before starting off with the solution, let me clarify you the use of __init__.py file. It just tells the python interpreter that the *.py files present there are importable which means they are modules and are/maybe a part of a package.
So, If you have N no of sub directories you have to put __init__.py file in all those sub directories such that they can also be imported. Inside __init__.py file you can also add some additional information like which path should be included, default functions,variables,scope,..etc. To know about these just google about __init__.py file or take some python library and go through the same __init__.py file to know about it. (Here lies the solution)
More Info:
modules
Be pythonic
So as stated by #Sushant Chaudhary your project structure should be like
proj-dir
--|--__init__.py
--package1
--|--__init__.py
--|--module1.py
--package2
--|--__init__.py
--|--module2.py
So now, If I put __init__.py file under my directory like above, Will
it be importable and work fine?
yes and no.
Yes :
If you are importing the modules within that project/package directory.
for example in your case
you are importing package1.module1 in pakage2.module2 as from package1 import module1.
Here you have to import the base dir inside the sub modules, Why? the project will run fine if you are running the module from the same place. i.e: inside package2 as python module2.py, But will throw ModuleNotFoundError If you run the module from some other directory. i.e: any other path except under package2 for example under proj-dir as python package2/module2.py. This is what happening in your case. You are running the module from project-dir.
So How to fix this?
1- You have to append basedir path to system path in module2.py as
from sys import path
dir_path = "/absolute/path/to/proj-dir"
sys.path.insert(0, dir_path)
So that module2 will be able to find package1 (and module1 inside it).
2- You have to add all the sub module paths in __init__.py file under proj-dir.
For example:
#__init__.py under lxml
# this is a package
def get_include():
"""
Returns a list of header include paths (for lxml itself, libxml2
and libxslt) needed to compile C code against lxml if it was built
with statically linked libraries.
"""
import os
lxml_path = __path__[0]
include_path = os.path.join(lxml_path, 'includes')
includes = [include_path, lxml_path]
for name in os.listdir(include_path):
path = os.path.join(include_path, name)
if os.path.isdir(path):
includes.append(path)
return includes
This is the __init__.py file of lxml (a python library for parsing html,xml data). You can refer any __init__.py file under any python libraries having sub modules.ex (os,sys). Here I've mentioned lxml because I thought it will be easy for you to understand. You can even check __init__.py file under other libraries/packages. Each will have it's own way of defining the path for submodules.
No
If you are trying to import modules outside the directory. Then you have to export the module path such that other modules can find them into environment variables. This can be done directly by appending absolute path of the base dir to PYTHONPATH or to PATH.
To know more:
PATH variables in OS
PYTHONPATH variable
So to solve your problem, include the paths to all the sub modules in __init__.py file under proj-dir and add the /absolute/path/to/proj-dir either to PYTHONPATH or PATH.
Hope the answer explains you about usage of __init__.py and solves your problem.
On Linux, you can just add the path to the parent folder of your src directory to ~/.local/lib/python3.6/site-packages/my_modules.pth. See
Using .pth files. You can then import modules in src from anywhere on your system.
NB1: Replace python3.6 by any version of Python you want to use.
NB2: If you use Python2.7 (don't know for other versions), you will need to create __init__.py (empty) files in src/ and src/utils.
NB3: Any name.pth file is ok for my_modules.pth.
Yes, you can only import code from installed packages or from files in you working directory or subdirectories.
the way I see it, your problem would be solved if you would have your module or package installed, like an yother package one installs and then imports (numpy, xml, json etc.)
I also have a package I constantly use in all my projects, ulitilies, and I know it's a pain with the importing.
here is a description on how to How to package a python application to make it pip-installable:
https://marthall.github.io/blog/how-to-package-a-python-app/
Navigate to your python installation folder
Navigate to lib
Navigate to site-packages
Make a new file called any_thing_you_want.pth
Type .../src/utils/helpers.py inside that file with your favorite text editor
Note: the ellipsis before scr/utils/helpers.py will look something like: C:/Users/blahblahblah/python_folders/scr... <- YOU DO NEED THIS!
This is a cheap way out but it keeps code clean, and is the least complicated. The downside is, for every folder your modules are in, example.pth will need them. Upside: works with Windows all the way up to Windows 10
I can't make this work..
My structure is:
program_name/
__init__.py
setup.py
src/
__init__.py
Process/
__init__.py
thefile.py
tests/
__init__.py
thetest.py
thetest.py:
from ..src.Process.thefile.py import sth
Running: pytest ./tests/thetest.py from program_name gives :
ValueError: attempted relative import beyond top-level package
I tried also other approaches but i am receiving various errors.
But I would expect for the above to work.
ValueError: Attempted relative import in non-package
States that you're trying to use relative import in the module, which are to be used for packages i.e. to make it a package add __init__.py and call the thetest.py from some file outside the package.
Directly running thetest.py from interpreter won't work.
Relative imports require that the module which uses them is being
imported itself either as package module.
Suggestion 1:
The current tests directory has a __init__.py file but that doesn't allow you to run it as a module (via shell) - to make your current (relative) import work, you need to import it in an external (to package) file/module - let's create a main.py (can name it anything you like):
main.py
program_name/
__init__.py
setup.py
src/
__init__.py
Process/
__init__.py
thefile.py
tests/
__init__.py
thetest.py
src/Process/thefile.py:
s = 'Hello world'
tests/thetest.py:
from ..src.Process.thefile import s
print s
main.py:
from program_name.tests.thetest import s
Executing main.py:
[nahmed#localhost ~]$ python main.py
Hello world
Suggestion 2:
Execute the file just above root dir i.e. one level up the program_name/ , in the following fashion:
[nahmed#localhost ~]$ python -m program_name.tests.thetest
Hell World
P.S. relative imports are for packages, not modules.
Just solved a similar problem with a lot of googling.
Here's two solutions without changing the existing file structor:
1
The way to import module from parent folder from ..src.Process.thefile.py import sth is called "relative import".
It's only supported when launching as a package from the top-level package. In your case, that is launching command line from the directory which contains program_name/ and type (for win environment)
python -m program_name.tests.thetest
or simply (useful for many pytest files):
python -m pytest
2
Otherwise -- when trying to run a script alone or from a non top-level package --
you could manually add directory to the PYTHONPATH at run time.
import sys
from os import path
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
from src.Process.thefile import s
Try the first one first see if it's compatiable with the pytest framework. Otherwise the second one should always solve the problem.
Reference (How to fix "Attempted relative import in non-package" even with __init__.py)
When importing a file, Python only searches the current directory, the directory that the entry-point script is running from.
you can use sys.path to include different locations
import sys
sys.path.insert(0, '/path/to/application/app/folder')
import thefile
Relative import not working properly in python2.6.5 getting "ValueError: Attempted relative import in non-package".
I am having all those __init__.py in proper place.
I have seen that error before when running a script that is actually inside a package. To the interpreter, it appears as though the package is not a package.
Try taking the script into another directory, putting your package inside your pythonpath, and import absolutely. Then, relative imports inside your package will work. NOTE: you can STILL not relatively import inside the end script - the easiest thing to do in this case is to make a "wrapper" script, that simply calls some entry point in your package.
You can go even further here by using setuptools to create a setup.py for your package, to make it distributable. Then, as a part of that, entry points would allow you to autogenerate scripts that called your package's code.
EDIT:
From your comment, it appears as though I wasn't quite clear. I'm not 100% sure of your directory structure because your comment above wasn't formatted, but I took it to be like this:
PythonEvent/
main.py
__init__.py
DBConnector/
__init__.py
connector.py
service/
__init__.py
myservice.py
When in myservice.py you have the line from ..DBConnector.connector import DBUpdate, the interpreter tries to import it relatively, UNLESS you are running myservice.py directly. This is what it appears you are doing.
Try making another dummy script outside of PythonEvent/ that is simply as follows:
from PythonEvent.service import myservice
if __name__ == '__main__':
myservice.main() # or whatever the entry point is called in myservice.
Then, set your PYTHONPATH environment variable to point to the parent directory of PythonEvent/ (or move PythonEvent/ to your site-packages).
main.py
setup.py
Main Package/ ->
__init__.py
subpackage_a/ ->
__init__.py
module_a.py
subpackage_b/ ->
__init__.py
module_b.py
i)
1.You run python main.py
2.main.py does: import app.package_a.module_a
3.module_a.py does import app.package_b.module_b
ii)
Alternatively 2 or 3 could use: from app.package_a import module_a
That will work as long as you have app in your PYTHONPATH. main.py could be anywhere then.
So you write a setup.py to copy (install) the whole app package and subpackages to the target system's python folders, and main.py to target system's script folders.
Thanks to https://stackoverflow.com/a/1083169