I understand there are many SO questions on relative imports. I will document the extent I have tried the solutions therein, but I am still unable to solve my problem.
I have the following directory structure. It's not mine by design but I'm game for modifying things as necessary (forked repo).
exp
main_Exp.py
kaffe
__init__.py
tensorflow
__init__.py
network_shape.py
ResNet
__init__.py (*)
ThreeDMM_shape.py
To run the model in this repo, I am to use
python main_Exp.py input_file.csv
Inside main_Exp.py:
sys.path.append('./kaffe')
sys.path.append('./ResNet')
from ThreeDMM_shape import ResNet_101 as resnet101_shape
from ThreeDMM_expr import ResNet_101 as resnet101_expr
Inside ResNet/ThreeDMM_shape.py:
sys.path.append('/home/usc/Desktop/Research/FG18/ExpNet_Code_Release/kaffe/tensorflow')
from network_shape import Network_Shape
Ok, so obviously I need to change this hard-coded absolute path. I'd like to do it the right way and not use my own specific path that I happened to install these files to.
So I try
from ..kaffee.tensorflow import Network_Shape
>>> ValueError: Attempted relative import in non-package
(1) I added __init__.py file in the ResNet folder (shown with the (*))
(2) I tried running the file as a module: python -m main_Exp input_file.csv
(3) I also tried adding __init__.py to the top level folder (exp), though I believe doing so is nonsense.
(4) Given that the first import was happening using the kaffe path that was appended to sys.path, I tried changing import to from .tensorflow.network_shape import Network_Shape
Same error after all steps.
So I'm not understanding the rules around relative imports and how to reference files in a sane way. I would really appreciate a pointer that helps me understand how to do this, and how to think about such imports in general!
exp is indeed not a package and won't be made to a package even if you add an init file to it, cause that won't magically add it to the paths python looks for packages. If you do add the __init__.py you can then run as python -m exp.main_Exp input_file.csv (from ../exp). This would make python recognize exp as a package and kaffe/ResNet as subpackages. You would need to change imports to from ResNet.ThreeDMM_shape import ResNet_101 as resnet101_shape etc.
Edit in response to comment:
Running from the parent dir using the m switch is the recommended way of running the script see for instance https://stackoverflow.com/a/23540051/281545 (that's for python 3 however it should still apply). If you want to avoid it (it would break hardcoded relative paths for one) you should add exp to sys path (once maybe is enough) then change the imports to absolute ones as in:
# main_Exp.py
sys.path.append(os.path.abspath(os.path.dirname(__file__))) # the exp folder
from ResNet.ThreeDMM_shape import ResNet_101 as resnet101_shape
from ResNet.ThreeDMM_expr import ResNet_101 as resnet101_expr
# ResNet/ThreeDMM_shape.py
from kaffee.tensorflow import Network_Shape
With the folder structure
lib/
abcd/
__init.py__
lib.py
app.py
the code
from lib.abcd import lib
works. But with this file structure:
bin/
app.py
lib/
abcd/
__init.py__
lib.py
the code
from ..lib.abcd import lib
gives an import error.
How to do the import properly when the library is in a sibling of the current folder? (or subfolder of a sibling folder)
I know that there might some hack that involves adding lib/ to the PATH, but is there an elegant Pythonic solution?
If not, is there a real internal reason to prevent users to do this simple import in a simple way?
Methods to do this
Method #1: Using the sys module: You can easily accomplish what you are trying to do using the sys module. To import the lib package, you can use one of the two codes listed below:
import sys
sys.path.append('<PATH_TO_LIB_FOLDER>')
from lib.abcd import lib
or
import sys
sys.path.insert(0, '<PATH_TO_LIB_FOLDER>')
Method #2: Using the os module: Another method is to use the os module. Here is an example code that imports the lib module using the os module by invoking the os.path.join method:
import os
path = os.path.join("<PATH>/lib/abcd", "lib")
from lib.abcd import lib
Method #3: Add the module to your PYTHONPATH: This is not the best method in most cases, but if you don't want to keep using the sys or os module to import lib, this is ideal. All you have to do is type this in your bash terminal:
export PYTHONPATH=<PATH_TO_LIB> python lib.py
Then in your python shell you can import it like this:
from lib.abcd import lib
Method #4: Combine the sys and os module (recommended): This is the most efficient method and will save you a lot of time. This code combines the os and sys module like this:
import sys, os
sys.path.append(os.path.abspath(os.path.join('..', 'lib')))
Then you can import your module with ease like this:
from lib.abcd import lib
How all the codes work:
All the codes above are very simple. All the examples except for "Method #3", add your module to the PYTHONPATH temporarily. "Method #3" on the other hand, adds the module to your PYTHONPATH permanently.
Surface level look
Normally, you can't. When importing a file, Python only searches the current directory, the directory that the entry-point script is running from, and sys.path which includes locations such as the package installation directory (it's actually a little more complex than this, but this covers most cases).
The reason you don't see this problem for importing installed modules is because they are installed in a location that is already on your path, or the location is added to the path by the installing utility (pip for example).
You can add to the Python path at runtime:
import sys
sys.path.insert(0, '../lib')
import file
You can also use sys.path.append('../lib'), but then it will be searched for last in your path and may be overridden by previous path entries.
I have read through the import documentation extensively, and as far as I can tell, the answer to your question is no, there is no way to do this in a purely "Pythonic" way.
More in depth look:
Looking deeper into the import documentation explains this:
The import statement combines two operations; it searches for the named module, then it binds the results of that search to a name in the local scope. The search operation of the import statement is defined as a call to the __import__() function, with the appropriate arguments. The return value of __import__() is used to perform the name binding operation of the import statement.
Looking more closely at __import__:
__import__(name, globals=None, locals=None, fromlist=(), level=0)
Note: This is an advanced function that is not needed in everyday Python programming, unlike importlib.import_module().
This function is invoked by the import statement. It can be replaced (by importing the builtins module and assigning to builtins.__import__) in order to change semantics of the import statement, but doing so is strongly discouraged as it is usually simpler to use import hooks (see PEP 302) to attain the same goals and does not cause issues with code which assumes the default import implementation is in use. Direct use of __import__() is also discouraged in favor of importlib.import_module().
level specifies whether to use absolute or relative imports. 0 (the default) means only perform absolute imports. Positive values for level indicate the number of parent directories to search relative to the directory of the module calling __import__() (see PEP 328 for the details).
This makes me think that you can specify a level in some way that may make the import automatically look on the parent path. I'm guessing that when import is called from your app.py that is not in it's own directory, level is set to 0 and it searches the same level and deeper.
When you call import in app.py from a subfolder, level is still set to 0 and thus can't find the other directories above it. I am investigating a "Pythonic" way of setting this level to 1 when running your script which would appear to fix this problem.
I have trouble importing package.
My file structure is like this:
filelib/
__init__.py
converters/
__init__.py
cmp2locus.py
modelmaker/
__init__.py
command_file.py
In module command_file.py I have a class named CommandFile which i want to call in the cmp2locus.py module.
I have tried the following in cmp2locus.py module:
import filelib.modelmaker.command_file
import modelmaker.command_file
from filelib.modelmaker.command_file import CommandFile
All these options return ImportError: No modules named ...
Appreciate any hint on solving this. I do not understand why this import does not work.
To perform these imports you have 3 options, I'll list them in the order I'd prefer. (For all of these options I will be assuming python 3)
Relative imports
Your file structure looks like a proper package file structure so this should work however anyone else trying this option should note that it requires you to be in a package; this won't work for some random script.
You'll also need to run the script doing the importing from outside the package, for example by importing it and running it from there rather than just running the cmp2locus.py script directly
Then you'll need to change your imports to be relative by using ..
So:
import filelib.modelmaker.command_file
becomes
from ..modelmaker import command_file
The .. refers to the parent folder (like the hidden file in file systems).
Also note you have to use the from import syntax because names starting with .. aren't valid identifiers in python. However you can of course import it as whatever you'd like using from import as.
See also the PEP
Absolute imports
If you place your package in site-packages (the directories returned by site.getsitepackages()) you will be able to use the format of imports that you were trying to use in the question. Note that this requires any users of your package to install it there too so this isn't ideal (although they probably would, relying on it is bad).
Modifying the python path
As Meera answered you can also directly modify the python path by using sys.
I dislike this option personally as it feels very 'hacky' but I've been told it can be useful as it gives you precise control of what you can import.
To import from another folder, you have to append that path of the folder to sys.path:
import sys
sys.path.append('path/filelib/modelmaker')
import command_file
Imagine this directory structure:
app/
__init__.py
sub1/
__init__.py
mod1.py
sub2/
__init__.py
mod2.py
I'm coding mod1, and I need to import something from mod2. How should I do it?
I tried from ..sub2 import mod2 but I'm getting an "Attempted relative import in non-package".
I googled around but found only "sys.path manipulation" hacks. Isn't there a clean way?
Edit: all my __init__.py's are currently empty
Edit2: I'm trying to do this because sub2 contains classes that are shared across sub packages (sub1, subX, etc.).
Edit3: The behaviour I'm looking for is the same as described in PEP 366 (thanks John B)
Everyone seems to want to tell you what you should be doing rather than just answering the question.
The problem is that you're running the module as '__main__' by passing the mod1.py as an argument to the interpreter.
From PEP 328:
Relative imports use a module's __name__ attribute to determine that module's position in the package hierarchy. If the module's name does not contain any package information (e.g. it is set to '__main__') then relative imports are resolved as if the module were a top level module, regardless of where the module is actually located on the file system.
In Python 2.6, they're adding the ability to reference modules relative to the main module. PEP 366 describes the change.
Update: According to Nick Coghlan, the recommended alternative is to run the module inside the package using the -m switch.
Here is the solution which works for me:
I do the relative imports as from ..sub2 import mod2
and then, if I want to run mod1.py then I go to the parent directory of app and run the module using the python -m switch as python -m app.sub1.mod1.
The real reason why this problem occurs with relative imports, is that relative imports works by taking the __name__ property of the module. If the module is being directly run, then __name__ is set to __main__ and it doesn't contain any information about package structure. And, thats why python complains about the relative import in non-package error.
So, by using the -m switch you provide the package structure information to python, through which it can resolve the relative imports successfully.
I have encountered this problem many times while doing relative imports. And, after reading all the previous answers, I was still not able to figure out how to solve it, in a clean way, without needing to put boilerplate code in all files. (Though some of the comments were really helpful, thanks to #ncoghlan and #XiongChiamiov)
Hope this helps someone who is fighting with relative imports problem, because going through PEP is really not fun.
main.py
setup.py
app/ ->
__init__.py
package_a/ ->
__init__.py
module_a.py
package_b/ ->
__init__.py
module_b.py
You run python main.py.
main.py does: import app.package_a.module_a
module_a.py does import app.package_b.module_b
Alternatively 2 or 3 could use: from app.package_a import module_a
That will work as long as you have app in your PYTHONPATH. main.py could be anywhere then.
So you write a setup.py to copy (install) the whole app package and subpackages to the target system's python folders, and main.py to target system's script folders.
"Guido views running scripts within a package as an anti-pattern" (rejected
PEP-3122)
I have spent so much time trying to find a solution, reading related posts here on Stack Overflow and saying to myself "there must be a better way!". Looks like there is not.
This is solved 100%:
app/
main.py
settings/
local_setings.py
Import settings/local_setting.py in app/main.py:
main.py:
import sys
sys.path.insert(0, "../settings")
try:
from local_settings import *
except ImportError:
print('No Import')
explanation of nosklo's answer with examples
note: all __init__.py files are empty.
main.py
app/ ->
__init__.py
package_a/ ->
__init__.py
fun_a.py
package_b/ ->
__init__.py
fun_b.py
app/package_a/fun_a.py
def print_a():
print 'This is a function in dir package_a'
app/package_b/fun_b.py
from app.package_a.fun_a import print_a
def print_b():
print 'This is a function in dir package_b'
print 'going to call a function in dir package_a'
print '-'*30
print_a()
main.py
from app.package_b import fun_b
fun_b.print_b()
if you run $ python main.py it returns:
This is a function in dir package_b
going to call a function in dir package_a
------------------------------
This is a function in dir package_a
main.py does: from app.package_b import fun_b
fun_b.py does from app.package_a.fun_a import print_a
so file in folder package_b used file in folder package_a, which is what you want. Right??
def import_path(fullpath):
"""
Import a file with full path specification. Allows one to
import from anywhere, something __import__ does not do.
"""
path, filename = os.path.split(fullpath)
filename, ext = os.path.splitext(filename)
sys.path.append(path)
module = __import__(filename)
reload(module) # Might be out of date
del sys.path[-1]
return module
I'm using this snippet to import modules from paths, hope that helps
This is unfortunately a sys.path hack, but it works quite well.
I encountered this problem with another layer: I already had a module of the specified name, but it was the wrong module.
what I wanted to do was the following (the module I was working from was module3):
mymodule\
__init__.py
mymodule1\
__init__.py
mymodule1_1
mymodule2\
__init__.py
mymodule2_1
import mymodule.mymodule1.mymodule1_1
Note that I have already installed mymodule, but in my installation I do not have "mymodule1"
and I would get an ImportError because it was trying to import from my installed modules.
I tried to do a sys.path.append, and that didn't work. What did work was a sys.path.insert
if __name__ == '__main__':
sys.path.insert(0, '../..')
So kind of a hack, but got it all to work!
So keep in mind, if you want your decision to override other paths then you need to use sys.path.insert(0, pathname) to get it to work! This was a very frustrating sticking point for me, allot of people say to use the "append" function to sys.path, but that doesn't work if you already have a module defined (I find it very strange behavior)
Let me just put this here for my own reference. I know that it is not good Python code, but I needed a script for a project I was working on and I wanted to put the script in a scripts directory.
import os.path
import sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
As #EvgeniSergeev says in the comments to the OP, you can import code from a .py file at an arbitrary location with:
import imp
foo = imp.load_source('module.name', '/path/to/file.py')
foo.MyClass()
This is taken from this SO answer.
Take a look at http://docs.python.org/whatsnew/2.5.html#pep-328-absolute-and-relative-imports. You could do
from .mod1 import stuff
From Python doc,
In Python 2.5, you can switch import‘s behaviour to absolute imports using a from __future__ import absolute_import directive. This absolute- import behaviour will become the default in a future version (probably Python 2.7). Once absolute imports are the default, import string will always find the standard library’s version. It’s suggested that users should begin using absolute imports as much as possible, so it’s preferable to begin writing from pkg import string in your code
I found it's more easy to set "PYTHONPATH" enviroment variable to the top folder:
bash$ export PYTHONPATH=/PATH/TO/APP
then:
import sub1.func1
#...more import
of course, PYTHONPATH is "global", but it didn't raise trouble for me yet.
On top of what John B said, it seems like setting the __package__ variable should help, instead of changing __main__ which could screw up other things. But as far as I could test, it doesn't completely work as it should.
I have the same problem and neither PEP 328 or 366 solve the problem completely, as both, by the end of the day, need the head of the package to be included in sys.path, as far as I could understand.
I should also mention that I did not find how to format the string that should go into those variables. Is it "package_head.subfolder.module_name" or what?
You have to append the module’s path to PYTHONPATH:
export PYTHONPATH="${PYTHONPATH}:/path/to/your/module/"
A hacky way to do it is to append the current directory to the PATH at runtime as follows:
import pathlib
import sys
sys.path.append(pathlib.Path(__file__).parent.resolve())
import file_to_import # the actual intended import
In contrast to another solution for this question this uses pathlib instead of os.path.
This method queries and auto populates the path:
import os
import inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
os.sys.path.insert(1, parentdir)
# print("currentdir = ", currentdir)
# print("parentdir=", parentdir)
What a debate!
Relative newcomer to python (but years of programming experience, and dislike of perl). Relative lay-person when it comes to the dark art of Apache setup, but I know what I (think I) need to get my little experimental projects working at home.
Here is my summary of what the situ seems to be.
If I use the -m 'module' approach, I need to:-
dot it all together;
run it from a parent folder;
lose the '.py';
create an empty (!) __init__.py file in every sub-folder.
How does that work in a cgi environment, where I have aliased my scripts directory, and want to run a script directly as /dirAlias/cgi_script.py??
Why is amending sys.path a hack? The python docs page states: "A program is free to modify this list for its own purposes." If it works, it works, right? The bean counters in Accounts don't care how it works.
I just want to go up one level and down into a 'modules' dir:-
.../py
/cgi
/build
/modules
so my 'modules' can be imported from either the cgi world or the server world.
I've tried the -m/modules approach but I think I prefer the following (and am not confused how to run it in cgi-space):-
Create XX_pathsetup.py in the /path/to/python/Lib dir (or any other dir in the default sys.path list). 'XX' is some identifier that declares an intent to setup my path according to the rules in the file.
In any script that wants to be able to import from the 'modules' dir in above directory config, simply import XX_pathsetup.py.
And here's my really simple XX_pathsetup.py:
import sys, os
pypath = sys.path[0].rsplit(os.sep,1)[0]
sys.path.insert( 0, pypath+os.sep+'modules' )
Not a 'hack', IMHO. 1 small file to put in the python 'Lib' dir, one import statement which declares intent to modify the path search order.
This question already has answers here:
Python relative import causes SyntaxError exception
(2 answers)
Closed 6 years ago.
The file structure of the code I am working with right now is as follows:
school_project/
__init__.py #(empty)
main_functions/
__init__.py #(empty)
render.py
filter.py
helper_functions/
__init__.py #(empty)
string.py
utility.py
Currently, I need to use functions founded in utility.py in the file render.py. My first attempt at solving this problem was to do import ..helper_functions.utility in the file render.py.
Unfortunately, it was met with the following error message.
import ..helper_functions.utility
^
SyntaxError: invalid syntax
First off, I have no idea why this relative import is not working.
Secondly, should I just use an absolute import instead? In the form import school_project.helper_functions.utility? If so, would I then need to add the directory that school_project/ is currently in to PYTHONPATH? How would I do this?
Would I just modify my computer's PATH and PYTHONPATH will adapt accordingly? Or are they separate entities and the process is a bit more involved? Ive looked at other threads but they all seem to modify PYTHONPATH at run time in the python script itself, something I see as a giant potential origin of bugs in the future.
This is the way you should do it:
from ..helper_functions import utility
This will not work if you run your python program normally due to relative imports.
This is the way you are supposed to run it:
python3 -m helper_functions.utility
But it's somewhat verbose, and doesn't mix well with a shebang line like #!/usr/bin/env python3.
Although it's not unique. Your package structure is more complex. You'll need to include the directory containing your package directory in PYTHONPATH, and do it like this.
from mypackage.mymodule import as_int
You can also do this. But this is not recommanded for beginners.
You just frob the PYTHONPATH in code first with this...
import sys
import os
PACKAGE_PARENT = '..'
SCRIPT_DIR = os.path.dirname(os.path.realpath(os.path.join(os.getcwd(), os.path.expanduser(__file__))))
sys.path.append(os.path.normpath(os.path.join(SCRIPT_DIR, PACKAGE_PARENT)))
from mypackage.mymodule import as_int
Absolute imports are recommended in Pep 8:
Absolute imports are recommended, as they are usually more readable and tend to be better behaved (or at least give better error messages) if the import system is incorrectly configured (such as when a directory inside a package ends up on sys.path ):
As for the PYTHONPATH, setting it once per terminal session where you work on the project might be best for now. Once you continue on to use an IDE like pycharm, this will be managed for you automagically