If I have a small python project split into a main directory with 2 subdirs:
src/
run.py
subdir1/
__init__.py
module1.py
subdir2/
__init__.py
module2.py
In order to include module2 in module1 would I have to use the full include path: from subdir1.subdir2.module2 import Class2 (relative to the python file that is being run), or could I use from subdir2.module2 import Class2? (relative to the file that statement is in)
src/
run.py
subdir1/
__init__.py
module1.py
subdir2/
__init__.py
module2.py
How is it now best to include module2 in module1, I have tried relative to where the program is run, i.e. from subdir2.module2 import Class2 but this gives the error "No module named module2" I cant help thinking this is not the most portable way of doing things anyhow, would it not be better with a path relative to the file where the statement exists, especcially in the 1st example.
I'm assuming it wont mess things up if I call the code from another active directory, or will it? python dir/run.py
Thanks!
There are two ways to import module:
absolute import
relative import
Absolute import makes python to look for desired module in directories stored in sys.path
Relative import address to module relative to current module. You can use relative import inside python packages only.
for first case You can use relative import because subdir1 is a package
for the second one You can't use relative import because subdir1 and subdir2 are not in package.
according to pep-8:
Absolute imports are recommended, as they are usually more readable and tend to be better behaved (or at least give better error messages) if the import system is incorrectly configured (such as when a directory inside a package ends up on sys.path):
import mypkg.sibling
from mypkg import sibling
from mypkg.sibling import example
However, explicit relative imports are an acceptable alternative to absolute imports, especially when dealing with complex package layouts where using absolute imports would be unnecessarily verbose:
from . import sibling
from .sibling import example
Standard library code should avoid complex package layouts and always use absolute imports.
Implicit relative imports should never be used and have been removed in Python 3.
Related
Consider the following directory
myProject
myCode.py
__init__.py
myProject2
__init__.py
myProject2Inner
myCode.py
__init__.py
myLibrary
__init__.py
myPackage1
__init__.py
myPackage1Code.py
myPackage2
__init__.py
myPackage2Code.py
If myCode.py is dependent on myPackage1Code.py and myPackage1Code.py is dependent on myPackage2Code.py
I am currently doing the following
sys.path.append(os.path.abspath('../myLibrary/myPackage2/'))
import myPackage2Code
in myPackage1Code.py to make the code run successfully. But this is obviously really bad since the library import path is entirely dependent on who is using it. For example if myProject2Inner requires myPackage1 then the code above wouldn't work.
I would have to do
sys.path.append(os.path.abspath('../../myLibrary/myPackage2/'))
import myPackage2Code
I think I am doing something really wrong here, can someone point me a direction of how to handle import path within a self containing library?
In your case, myLibrary, myPackage1 and myPackage2 are packages. To import modules (or packages) from other packages, you must either use an absolute or relative path:
# in myPackage1Code.py
# absolute import
from myLibrary.myPackage2 import myPackage2Code
# relative import
from ..myPackage2 import myPackage2Code
This uniquely identifies the module you actually want, and tells Python where to find it. Note that . and .. are not file-system operations: they also work with dynamically composed namespace packages.
If you want to execute a script contained inside your package, you execute it as part of the package:
python2 -m myLibrary.myPackage1.myPackage1Code
Python2 also has implicit relative imports:
# in myLibrary/__init__.py
from myPackage2 import myPackage2Code
This form is generally discouraged, as it breaks if there is a global myPackage2. It also does not work with Python3.
Note that for packages to work, you have to use them as such! If you directly access part of a package (don't do this at home!)
# directly run code module of a package in the shell
python2 myLibrary/myPackage1/myPackage1Code.py
# directly import module of a package
sys.path.append(os.path.abspath('../../myLibrary/myPackage2/'))
import myPackage2Code
then Python does not know that myPackage2Code belongs to myLibrary.myPackage2.
This has two notable effects:
The myPackage2Code cannot use relative imports. Python considers it a top-level module, so imports cannot go "up" in the package hierarchy.
If another module imports it with its full path, this creates two separate modules myPackage2Code and myLibrary.myPackage2.myPackage2Code. Since these contain separate objects, they for example fail isinstance checks of except clauses.
I'm looking for a clarification of the following statement in PEP 8:
Implicit relative imports should never be used and have been removed in Python 3.
Suppose that in Python 2 I have the following directory structure (in a directory called test):
.
├── test_recurring_interval.py
└── test_utils.py
In the file test_recurring_interval.py, I have a line
import test_utils
Is this an implicit relative import? If so, how should I change it to make it explicit?
Whether this is an implicit relative import depends on whether the module containing the import statement is part of a package. This is trickier than it might at first seem, since whether a module is part of a package depends on how Python was executed and how __package__ and sys.path are set.
It doesn't look like there are any packages involved here. This is an absolute import, then, not a relative import.
An explicit relative import is explicitly relative when it starts with a ..
Implicitily relative:
import test_utils
Explicitily relative:
import .test_utils
This question already has answers here:
Python3 correct way to import relative or absolute?
(2 answers)
Closed 2 years ago.
I have the following directory:
mydirectory
├── __init__.py
├── file1.py
└── file2.py
I have a function f defined in file1.py.
If, in file2.py, I do
from .file1 import f
I get the following error:
SystemError: Parent module '' not loaded, cannot perform relative
import
Why? And how to make it work?
Launching modules inside a package as executables is a bad practice.
When you develop something you either build a library, which is intended to be imported by other programs and thus it doesn't make much sense to allow executing its submodules directly, or you build an executable in which case there's no reason to make it part of a package.
This is why in setup.py you distinguish between packages and scripts. The packages will go under site-packages while the scripts will be installed under /usr/bin (or similar location depending on the OS).
My recommendation is thus to use the following layout:
/
├── mydirectory
| ├── __init__.py
| ├── file1.py
└── file2.py
Where file2.py imports file1.py as any other code that wants to use the library mydirectory, with an absolute import:
from mydirectory.file1 import f
When you write a setup.py script for the project you simply list mydirectory as a package and file2.py as a script and everything will work. No need to fiddle with sys.path.
If you ever, for some reason, really want to actually run a submodule of a package, the proper way to do it is to use the -m switch:
python -m mydirectory.file1
This loads the whole package and then executes the module as a script, allowing the relative import to succeed.
I'd personally avoid doing this. Also because a lot of people don't even know you can do this and will end up getting the same error as you and think that the package is broken.
Regarding the currently accepted answer, which says that you should just use an implicit relative import from file1 import f because it will work since they are in the same directory:
This is wrong!
It will not work in python3 where implicit relative imports are disallowed and will surely break if you happen to have installed a file1 module (since it will be imported instead of your module!).
Even if it works the file1 will not be seen as part of the mydirectory package. This can matter.
For example if file1 uses pickle, the name of the package is important for proper loading/unloading of data.
When launching a python source file, it is forbidden to import another file, that is in the current package, using relative import.
In documentation it is said:
Note that relative imports are based on the name of the current module. Since the name of the main module is always "__main__", modules intended for use as the main module of a Python application must always use absolute imports.
So, as #mrKelley said, you need to use absolute import in such situation.
since file1 and file2 are in the same directory, you don't even need to have an __init__.py file. If you're going to be scaling up, then leave it there.
To import something in a file in the same directory, just do like this
from file1 import f
i.e., you don't need to do the relative path .file1 because they are in the same directory.
If your main function, script, or whatever, that will be running the whole application is in another directory, then you will have to make everything relative to wherever that is being executed.
myproject/
mypackage
├── __init__.py
├── file1.py
├── file2.py
└── file3.py
mymainscript.py
Example to import from one file to another
#file1.py
from myproject import file2
from myproject.file3 import MyClass
Import the package example to the mainscript
#mymainscript.py
import mypackage
https://docs.python.org/3/tutorial/modules.html#packages
https://docs.python.org/3/reference/import.html#regular-packages
https://docs.python.org/3/reference/simple_stmts.html#the-import-statement
https://docs.python.org/3/glossary.html#term-import-path
The variable sys.path is a list of strings that determines the interpreter’s search path for modules. It is initialized to a default path taken from the environment variable PYTHONPATH, or from a built-in default if PYTHONPATH is not set. You can modify it using standard list operations:
import sys
sys.path.append('/ufs/guido/lib/python')
sys.path.insert(0, '/ufs/guido/myhaxxlib/python')
Inserting it at the beginning has the benefit of guaranteeing that the path is searched before others (even built-in ones) in the case of naming conflicts.
I noticed I've made about 5 questions asking about relative imports, each with working solutions but with different situations. The more I read the docs and the more I try out different cases the more I'm getting confused and questioning whether I'm actually reading the words correctly.
From pep 328:
A single leading dot indicates a relative import,
starting with the current package. Two or more leading dots
give a relative import to the parent(s) of the current package,
one level per dot after the first.
My understanding is that "current package" is the keyword here. What IS the "current package"? Is it the package of the module where execution begins? (ie: the file with __ name __ == "__ main __").
Anyways, here is a simple situation of me using .. notation.
main/
lib/
__init__.py
myLib.py
plugin/
__init__.py
needLib.py
run.py
run.py imports needLib: from plugin import needLib
needLib imports myLib: from ..lib import myLib
What's going through my head: (needLib) goes up a package, go into lib, grab myLib
Here is how I execute: python run.py
Result: Attempted relative import beyond toplevel package
But when I changed the import call to from .lib import myLib, it works.
I don't understand why the latter works and the .. notation doesn't. Of course, main isn't a package, so how about I just throw in a init.py there as well, resulting in
main/
lib/
__init__.py
myLib.py
other/
__init__.py
needLib.py
__init__.py
run.py
But that didn't make a difference; still attempting to import beyond top-level package
Check out this question: python: forcing relative imports to search from script file
The .. notation actually worked! Now that I think about it, I don't understand why it works. Execution in that scenario started 2 levels up from where the relative import takes place, while execution in this scenario starts 1 level up from where the relative import takes place. Other than where I begin execution, the situation is identical.
Both are me saying "go up one directory, go into the lib package, and grab the module you want".
What is the key difference between this situation and the referenced situation? Why does .. work there, but not here? Does me using 2.6 have anything to do with this? Or perhaps how I am executing it? python run.py
As you found out, from ..lib import myLib does not work as main is not a package. Relative imports only work within a module's own package. In your example, lib and other are two distinct packages.
Putting an __init__.py in main only works if you move run outside of the newly created main package, so that the current directory (part of sys.path) is not within a package.
I know there are A LOT of similar or the same questions, but i still cannot understand / find the right way for me to work with modules. Python is my favorite language, and i like everything in it except working with imports: recursive imports (when you try to reference a name that is not yet there), import paths, etc.
So, I have this kind of a project structure:
my_project/
package1/
__init__.py
module1
module2
package2/
__init__.py
module1
module2
Package1 may be used a standalone unit, but is also expected to be imported by package2.
What am i doing now, is that, for example, in package1.module1 i write from package1 import module2, i.e. using full path to imported module. I do this because if i use import module2 -- this will not work when the module will be imported from another package (package2). I also cannot use from . import module2 -- this will not work when running module1 directly.
OK, so for from package1 import module2 in package1.module1 to work in both cases (when running directly package1.module1 and when importing it from package2) i add these lines at the beginning of package1.module1:
import os, sys
currDir = os.path.dirname(os.path.realpath(__file__))
rootDir = os.path.abspath(os.path.join(currDir, '..'))
if rootDir not in sys.path: # add parent dir to paths
sys.path.append(rootDir)
For me this works, but i feel this is not pythonic. Am i doing something wrong?
Should i, instead, always run package1.module1 from project root? If so, this makes inconvenient to run it from an IDE -- i need somehow to set paths in it.
UPDATE: I tried to add a file root.pth to package1 dir with contents of ... But it didn't work -- i guess it's intended for something else.
CONCLUSIONS:
Always use absolute imports: import package1.module1
Add a bootstrapper to the root folder to start some of the modules as a standalone script. This solves running the script form an IDE and is a pythonic approach.
On 4/22/07, Brett Cannon wrote:
This PEP is to change the if __name__ == "__main__": ... idiom to
if __name__ == sys.main: ... so that you at least have a chance
to execute module in a package that use relative imports.
Ran this PEP past python-ideas. Stopped the discussion there when too
many new ideas were being proposed. =) I have listed all of them in
the Rejected Ideas section, although if overwhelming support for one
comes forward the PEP can shift to one of them.
I'm -1 on this and on any other proposed twiddlings of the __main__
machinery. The only use case seems to be running scripts that happen
to be living inside a module's directory, which I've always seen as an
antipattern. To make me change my mind you'd have to convince me that
it isn't.
--Guido van Rossum
What is the entry point for your program? Usually the entry point for a program will be at the root of the project. Since it is at the root, all the modules within the root will be importable, provided there is an __init__.py file in them.
So, using your example:
my_project/
main.py
package1/
__init__.py
module1
module2
package2/
__init__.py
module1
module2
main.py would be the entry point for your program. Because the file that is executed as main is automatically put on the PYTHONPATH, both package1 and package2 are available from the top level import.
# in main.py
from package1.module1 import *
from package1.module2 import *
# in package1.module1
import module2
from package2.module1 import *
# in package2.module1 import *
import module2
from package1.module1 import *
Note that in the above, package1 and package2 depend on each other. That should never be the case. But this is just an example of being able to import from anywhere.
main.py doesn't have to be anything fancy either. It can be very simple:
# main.py
if __name__ == '__main__':
from package1.module1 import SomeClass
SomeClass().start()
The point I'm trying to make, is that if a module needs to be accessible by other modules, that module should be available as a top level import. A module should not attempt to put itself as a top level import (directly on the PYTHONPATH).
It should be the responsibility of the project for ensuring that all imports can be satisfied if the module is included directly in the project. There are two ways to do this. The first is by creating a bootstrapper file such as main.py in the project folder. The other, is by creating a file that adds all relevant paths to PYTHONPATH, that is loaded by any entry points that may exist.
For example:
# setup.py
import sys
def load():
paths = ['/path1/','/path2/','/path3/']
for p in path:
sys.path.insert(0, p)
# entrypoint.py
from setup import load
load()
# continue with program
The main thing to take away, is that a module is not supposed to put itself on the path. The path should be determined automatically by the entry point into the program, or defined explicitly by a setup script that knows where all the relevant modules are.
I generally create each package as an installable package (ie, create a setup.py file), and then install them into a virtualenv just for this project, using pip.
You can even install the using pip -e if they are still under development.
I'm 5 years late here.. but just do export PYTHONPATH=/path1:/path2: (notice trailing ":") - this way your working directory (from which you run python) will be in the path.