Setup
test/
main.py
pkg/
a.py
__init__.py
main.py contains:
import pkg
pkg.a
__init__.py contains:
from . import a
main.py can be run without errors.
Question
Changing the content of __init__.py to
import a
gives the following error when running main.py:
Traceback (most recent call last):
File "C:/Users/me/PycharmProjects/test/main.py", line 1, in <module>
import pkg
File "C:\Users\me\PycharmProjects\test\pkg\__init__.py", line 1, in <module>
import a
ModuleNotFoundError: No module named 'a'
Interestingly, __init__.py can be executed directly with python __init__.py without errors.
What's going on?
When you run a python script, it's parent folder is added to sys.path
run main.py: sys.path[0] = '../test'
run init.py: sys.path[0] = '../test/pkg'
Your case: You try to "absolute-like" import a in __init__.py but the parent folder of a.py - which is '../test/pkg' - is not in the sys.path when you run main.py. This is why you get an error. However, your absolute import is incomplete as it should always start at the top level folder, e.g.
from test.pkg import a
Final answer to your question: You don't have to use relative imports!
See: PEP-8: Absolute imports are recommended, as they are usually more readable and tend to be better behaved (or at least give better error messages) if the import system is incorrectly configured (such as when a directory inside a package ends up on sys.path).
And keep in mind that relative imports don't work in a top-level-script when __name__ = "__main__", but from imported modules only.
You can learn more about absolute and relative imports here:
Absolute vs. explicit relative import of Python module
https://realpython.com/absolute-vs-relative-python-imports/
I suppose you are using Pycharm? Then that's one of the confusion cause.
For example, let's say your directory looks like this
project1
p1.py
test/
__init__.py
main.py
pkg/
a.py
__init__.py
If you run (F10) the main.py your default working directory will be project1/test, which does not contain the a.py so import a will not find anything.
But if you run (F10) the pkg/__init__.py your working directory will be project1/test/pkg which has the a.py, and it works like what you tested.
So in these situation, if you use from . import a it will look for the directory that file is, project1/test/pkg in this case, which will always work regardless your working directory.
I have the following directory tree:
project/
A/
__init__.py
foo.py
TestA/
__init__.py
testFoo.py
the content of testFoo is:
import unittest
from A import foo
from the project directory I run python testA/testFoo.py
I get a ModuleNotFoundError No module named A
I have two question: how to improt and run A.foo from TestA.testFoo and why is it so difficult to grasp the import logic in Python? Isn't there any debug trick to solve this kind of issues rapidly, I'm sorry I have to bother you with such basics questions?
When your are executing a file an environment variable called python path is generated, python import work with this variable to find your file to import, this path is generated with the path of the file you are executing and it will search in the current directory and sub directories containing an __init__.py file, if you want to import from a directory on the same level you need to modify your python path or change the architecture of your project so the file executed is always on top level.
you can include path to your python path like this :
import sys
sys.path.insert(0, "/path/to/file.py")
You can read more on import system : https://docs.python.org/3/reference/import.html
The best way in my opinion is to not touch the python path and include your test directoy into the directory where tested files are:
project/
A/
__init__.py
foo.py
TestA/
__init__.py
testFoo.py
Then run the python -m unittest command into your A or project directory, it will search into your current and sub directories for test and execute it.
More on unittest here : https://docs.python.org/3/library/unittest.html
Add the folder project/testA to the system pythonpath first:
import sys
sys.path.insert(0, "/path/to/pythonfile")
and try the import again.
Can you try this ?
Create an empty file __init__.py in subdirectory TestA. And add at the begin of main code
from __future__ import absolute_import
Then import as below :
import A.foo as testfoo
The recommended way in py3 may be like below
echo $pwd
$ /home/user/project
python -m testA.testFoo
The way of execute module python -m in python is a good way to replace relative references。
You definitely cannot find A because python need look from sys.path, PYTHONPATH to find the module.
And python will automatically add current top level script to sys.path not currently directory to sys.path. So if you add print(sys.path) in testFoo.py, you will see it only add project/TestA to the sys.path.
Another word, the project did not be included in sys.path, then how python can find the module A?
So you had to add the project folder to sys.path by yourself, and, this just needed in top script, something like follows:
import unittest
import sys
import os
file_path = os.path.abspath(os.path.dirname(__file__)).replace('\\', '/')
lib_path = os.path.abspath(os.path.join(file_path, '..')).replace('\\', '/')
sys.path.append(lib_path)
I can't make this work..
My structure is:
program_name/
__init__.py
setup.py
src/
__init__.py
Process/
__init__.py
thefile.py
tests/
__init__.py
thetest.py
thetest.py:
from ..src.Process.thefile.py import sth
Running: pytest ./tests/thetest.py from program_name gives :
ValueError: attempted relative import beyond top-level package
I tried also other approaches but i am receiving various errors.
But I would expect for the above to work.
ValueError: Attempted relative import in non-package
States that you're trying to use relative import in the module, which are to be used for packages i.e. to make it a package add __init__.py and call the thetest.py from some file outside the package.
Directly running thetest.py from interpreter won't work.
Relative imports require that the module which uses them is being
imported itself either as package module.
Suggestion 1:
The current tests directory has a __init__.py file but that doesn't allow you to run it as a module (via shell) - to make your current (relative) import work, you need to import it in an external (to package) file/module - let's create a main.py (can name it anything you like):
main.py
program_name/
__init__.py
setup.py
src/
__init__.py
Process/
__init__.py
thefile.py
tests/
__init__.py
thetest.py
src/Process/thefile.py:
s = 'Hello world'
tests/thetest.py:
from ..src.Process.thefile import s
print s
main.py:
from program_name.tests.thetest import s
Executing main.py:
[nahmed#localhost ~]$ python main.py
Hello world
Suggestion 2:
Execute the file just above root dir i.e. one level up the program_name/ , in the following fashion:
[nahmed#localhost ~]$ python -m program_name.tests.thetest
Hell World
P.S. relative imports are for packages, not modules.
Just solved a similar problem with a lot of googling.
Here's two solutions without changing the existing file structor:
1
The way to import module from parent folder from ..src.Process.thefile.py import sth is called "relative import".
It's only supported when launching as a package from the top-level package. In your case, that is launching command line from the directory which contains program_name/ and type (for win environment)
python -m program_name.tests.thetest
or simply (useful for many pytest files):
python -m pytest
2
Otherwise -- when trying to run a script alone or from a non top-level package --
you could manually add directory to the PYTHONPATH at run time.
import sys
from os import path
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
from src.Process.thefile import s
Try the first one first see if it's compatiable with the pytest framework. Otherwise the second one should always solve the problem.
Reference (How to fix "Attempted relative import in non-package" even with __init__.py)
When importing a file, Python only searches the current directory, the directory that the entry-point script is running from.
you can use sys.path to include different locations
import sys
sys.path.insert(0, '/path/to/application/app/folder')
import thefile
This question already has answers here:
Python3 correct way to import relative or absolute?
(2 answers)
Closed 2 years ago.
I have the following directory:
mydirectory
├── __init__.py
├── file1.py
└── file2.py
I have a function f defined in file1.py.
If, in file2.py, I do
from .file1 import f
I get the following error:
SystemError: Parent module '' not loaded, cannot perform relative
import
Why? And how to make it work?
Launching modules inside a package as executables is a bad practice.
When you develop something you either build a library, which is intended to be imported by other programs and thus it doesn't make much sense to allow executing its submodules directly, or you build an executable in which case there's no reason to make it part of a package.
This is why in setup.py you distinguish between packages and scripts. The packages will go under site-packages while the scripts will be installed under /usr/bin (or similar location depending on the OS).
My recommendation is thus to use the following layout:
/
├── mydirectory
| ├── __init__.py
| ├── file1.py
└── file2.py
Where file2.py imports file1.py as any other code that wants to use the library mydirectory, with an absolute import:
from mydirectory.file1 import f
When you write a setup.py script for the project you simply list mydirectory as a package and file2.py as a script and everything will work. No need to fiddle with sys.path.
If you ever, for some reason, really want to actually run a submodule of a package, the proper way to do it is to use the -m switch:
python -m mydirectory.file1
This loads the whole package and then executes the module as a script, allowing the relative import to succeed.
I'd personally avoid doing this. Also because a lot of people don't even know you can do this and will end up getting the same error as you and think that the package is broken.
Regarding the currently accepted answer, which says that you should just use an implicit relative import from file1 import f because it will work since they are in the same directory:
This is wrong!
It will not work in python3 where implicit relative imports are disallowed and will surely break if you happen to have installed a file1 module (since it will be imported instead of your module!).
Even if it works the file1 will not be seen as part of the mydirectory package. This can matter.
For example if file1 uses pickle, the name of the package is important for proper loading/unloading of data.
When launching a python source file, it is forbidden to import another file, that is in the current package, using relative import.
In documentation it is said:
Note that relative imports are based on the name of the current module. Since the name of the main module is always "__main__", modules intended for use as the main module of a Python application must always use absolute imports.
So, as #mrKelley said, you need to use absolute import in such situation.
since file1 and file2 are in the same directory, you don't even need to have an __init__.py file. If you're going to be scaling up, then leave it there.
To import something in a file in the same directory, just do like this
from file1 import f
i.e., you don't need to do the relative path .file1 because they are in the same directory.
If your main function, script, or whatever, that will be running the whole application is in another directory, then you will have to make everything relative to wherever that is being executed.
myproject/
mypackage
├── __init__.py
├── file1.py
├── file2.py
└── file3.py
mymainscript.py
Example to import from one file to another
#file1.py
from myproject import file2
from myproject.file3 import MyClass
Import the package example to the mainscript
#mymainscript.py
import mypackage
https://docs.python.org/3/tutorial/modules.html#packages
https://docs.python.org/3/reference/import.html#regular-packages
https://docs.python.org/3/reference/simple_stmts.html#the-import-statement
https://docs.python.org/3/glossary.html#term-import-path
The variable sys.path is a list of strings that determines the interpreter’s search path for modules. It is initialized to a default path taken from the environment variable PYTHONPATH, or from a built-in default if PYTHONPATH is not set. You can modify it using standard list operations:
import sys
sys.path.append('/ufs/guido/lib/python')
sys.path.insert(0, '/ufs/guido/myhaxxlib/python')
Inserting it at the beginning has the benefit of guaranteeing that the path is searched before others (even built-in ones) in the case of naming conflicts.
How do I import a Python module given its relative path?
For example, if dirFoo contains Foo.py and dirBar, and dirBar contains Bar.py, how do I import Bar.py into Foo.py?
Here's a visual representation:
dirFoo\
Foo.py
dirBar\
Bar.py
Foo wishes to include Bar, but restructuring the folder hierarchy is not an option.
Assuming that both your directories are real Python packages (do have the __init__.py file inside them), here is a safe solution for inclusion of modules relatively to the location of the script.
I assume that you want to do this, because you need to include a set of modules with your script. I use this in production in several products and works in many special scenarios like: scripts called from another directory or executed with python execute instead of opening a new interpreter.
import os, sys, inspect
# realpath() will make your script run, even if you symlink it :)
cmd_folder = os.path.realpath(os.path.abspath(os.path.split(inspect.getfile( inspect.currentframe() ))[0]))
if cmd_folder not in sys.path:
sys.path.insert(0, cmd_folder)
# Use this if you want to include modules from a subfolder
cmd_subfolder = os.path.realpath(os.path.abspath(os.path.join(os.path.split(inspect.getfile( inspect.currentframe() ))[0],"subfolder")))
if cmd_subfolder not in sys.path:
sys.path.insert(0, cmd_subfolder)
# Info:
# cmd_folder = os.path.dirname(os.path.abspath(__file__)) # DO NOT USE __file__ !!!
# __file__ fails if the script is called in different ways on Windows.
# __file__ fails if someone does os.chdir() before.
# sys.argv[0] also fails, because it doesn't not always contains the path.
As a bonus, this approach does let you force Python to use your module instead of the ones installed on the system.
Warning! I don't really know what is happening when current module is inside an egg file. It probably fails too.
Be sure that dirBar has the __init__.py file -- this makes a directory into a Python package.
You could also add the subdirectory to your Python path so that it imports as a normal script.
import sys
sys.path.insert(0, <path to dirFoo>)
import Bar
import os
import sys
lib_path = os.path.abspath(os.path.join(__file__, '..', '..', '..', 'lib'))
sys.path.append(lib_path)
import mymodule
Just do simple things to import the .py file from a different folder.
Let's say you have a directory like:
lib/abc.py
Then just keep an empty file in lib folder as named
__init__.py
And then use
from lib.abc import <Your Module name>
Keep the __init__.py file in every folder of the hierarchy of the import module.
If you structure your project this way:
src\
__init__.py
main.py
dirFoo\
__init__.py
Foo.py
dirBar\
__init__.py
Bar.py
Then from Foo.py you should be able to do:
import dirFoo.Foo
Or:
from dirFoo.Foo import FooObject
Per Tom's comment, this does require that the src folder is accessible either via site_packages or your search path. Also, as he mentions, __init__.py is implicitly imported when you first import a module in that package/directory. Typically __init__.py is simply an empty file.
The easiest method is to use sys.path.append().
However, you may be also interested in the imp module.
It provides access to internal import functions.
# mod_name is the filename without the .py/.pyc extention
py_mod = imp.load_source(mod_name,filename_path) # Loads .py file
py_mod = imp.load_compiled(mod_name,filename_path) # Loads .pyc file
This can be used to load modules dynamically when you don't know a module's name.
I've used this in the past to create a plugin type interface to an application, where the user would write a script with application specific functions, and just drop thier script in a specific directory.
Also, these functions may be useful:
imp.find_module(name[, path])
imp.load_module(name, file, pathname, description)
This is the relevant PEP:
http://www.python.org/dev/peps/pep-0328/
In particular, presuming dirFoo is a directory up from dirBar...
In dirFoo\Foo.py:
from ..dirBar import Bar
The easiest way without any modification to your script is to set PYTHONPATH environment variable. Because sys.path is initialized from these locations:
The directory containing the input script (or the current
directory).
PYTHONPATH (a list of directory names, with the same
syntax as the shell variable PATH).
The installation-dependent default.
Just run:
export PYTHONPATH=/absolute/path/to/your/module
You sys.path will contains above path, as show below:
print sys.path
['', '/absolute/path/to/your/module', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client', '/usr/lib/python2.7/dist-packages/ubuntuone-client', '/usr/lib/python2.7/dist-packages/ubuntuone-control-panel', '/usr/lib/python2.7/dist-packages/ubuntuone-couch', '/usr/lib/python2.7/dist-packages/ubuntuone-installer', '/usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol']
In my opinion the best choice is to put __ init __.py in the folder and call the file with
from dirBar.Bar import *
It is not recommended to use sys.path.append() because something might gone wrong if you use the same file name as the existing python package. I haven't test that but that will be ambiguous.
The quick-and-dirty way for Linux users
If you are just tinkering around and don't care about deployment issues, you can use a symbolic link (assuming your filesystem supports it) to make the module or package directly visible in the folder of the requesting module.
ln -s (path)/module_name.py
or
ln -s (path)/package_name
Note: A "module" is any file with a .py extension and a "package" is any folder that contains the file __init__.py (which can be an empty file). From a usage standpoint, modules and packages are identical -- both expose their contained "definitions and statements" as requested via the import command.
See: http://docs.python.org/2/tutorial/modules.html
from .dirBar import Bar
instead of:
from dirBar import Bar
just in case there could be another dirBar installed and confuse a foo.py reader.
For this case to import Bar.py into Foo.py, first I'd turn these folders into Python packages like so:
dirFoo\
__init__.py
Foo.py
dirBar\
__init__.py
Bar.py
Then I would do it like this in Foo.py:
from .dirBar import Bar
If I wanted the namespacing to look like Bar.whatever, or
from . import dirBar
If I wanted the namespacing dirBar.Bar.whatever. This second case is useful if you have more modules under the dirBar package.
Add an __init__.py file:
dirFoo\
Foo.py
dirBar\
__init__.py
Bar.py
Then add this code to the start of Foo.py:
import sys
sys.path.append('dirBar')
import Bar
Relative sys.path example:
# /lib/my_module.py
# /src/test.py
if __name__ == '__main__' and __package__ is None:
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../lib')))
import my_module
Based on this answer.
Well, as you mention, usually you want to have access to a folder with your modules relative to where your main script is run, so you just import them.
Solution:
I have the script in D:/Books/MyBooks.py and some modules (like oldies.py). I need to import from subdirectory D:/Books/includes:
import sys,site
site.addsitedir(sys.path[0] + '\\includes')
print (sys.path) # Just verify it is there
import oldies
Place a print('done') in oldies.py, so you verify everything is going OK. This way always works because by the Python definition sys.path as initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter.
If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH.
Another solution would be to install the py-require package and then use the following in Foo.py
import require
Bar = require('./dirBar/Bar')
Simply you can use: from Desktop.filename import something
Example:
given that the file is name test.py in directory
Users/user/Desktop , and will import everthing.
the code:
from Desktop.test import *
But make sure you make an empty file called "__init__.py" in that directory
Here's a way to import a file from one level above, using the relative path.
Basically, just move the working directory up a level (or any relative location), add that to your path, then move the working directory back where it started.
#to import from one level above:
cwd = os.getcwd()
os.chdir("..")
below_path = os.getcwd()
sys.path.append(below_path)
os.chdir(cwd)
I'm not experienced about python, so if there is any wrong in my words, just tell me. If your file hierarchy arranged like this:
project\
module_1.py
module_2.py
module_1.py defines a function called func_1(), module_2.py:
from module_1 import func_1
def func_2():
func_1()
if __name__ == '__main__':
func_2()
and you run python module_2.py in cmd, it will do run what func_1() defines. That's usually how we import same hierarchy files. But when you write from .module_1 import func_1 in module_2.py, python interpreter will say No module named '__main__.module_1'; '__main__' is not a package. So to fix this, we just keep the change we just make, and move both of the module to a package, and make a third module as a caller to run module_2.py.
project\
package_1\
module_1.py
module_2.py
main.py
main.py:
from package_1.module_2 import func_2
def func_3():
func_2()
if __name__ == '__main__':
func_3()
But the reason we add a . before module_1 in module_2.py is that if we don't do that and run main.py, python interpreter will say No module named 'module_1', that's a little tricky, module_1.py is right beside module_2.py. Now I let func_1() in module_1.py do something:
def func_1():
print(__name__)
that __name__ records who calls func_1. Now we keep the . before module_1 , run main.py, it will print package_1.module_1, not module_1. It indicates that the one who calls func_1() is at the same hierarchy as main.py, the . imply that module_1 is at the same hierarchy as module_2.py itself. So if there isn't a dot, main.py will recognize module_1 at the same hierarchy as itself, it can recognize package_1, but not what "under" it.
Now let's make it a bit complicated. You have a config.ini and a module defines a function to read it at the same hierarchy as 'main.py'.
project\
package_1\
module_1.py
module_2.py
config.py
config.ini
main.py
And for some unavoidable reason, you have to call it with module_2.py, so it has to import from upper hierarchy.module_2.py:
import ..config
pass
Two dots means import from upper hierarchy (three dots access upper than upper,and so on). Now we run main.py, the interpreter will say:ValueError:attempted relative import beyond top-level package. The "top-level package" at here is main.py. Just because config.py is beside main.py, they are at same hierarchy, config.py isn't "under" main.py, or it isn't "leaded" by main.py, so it is beyond main.py. To fix this, the simplest way is:
project\
package_1\
module_1.py
module_2.py
config.py
config.ini
main.py
I think that is coincide with the principle of arrange project file hierarchy, you should arrange modules with different function in different folders, and just leave a top caller in the outside, and you can import how ever you want.
This also works, and is much simpler than anything with the sys module:
with open("C:/yourpath/foobar.py") as f:
eval(f.read())
Call me overly cautious, but I like to make mine more portable because it's unsafe to assume that files will always be in the same place on every computer. Personally I have the code look up the file path first. I use Linux so mine would look like this:
import os, sys
from subprocess import Popen, PIPE
try:
path = Popen("find / -name 'file' -type f", shell=True, stdout=PIPE).stdout.read().splitlines()[0]
if not sys.path.__contains__(path):
sys.path.append(path)
except IndexError:
raise RuntimeError("You must have FILE to run this program!")
That is of course unless you plan to package these together. But if that's the case you don't really need two separate files anyway.