Okay, the scenario is very simple. I have this file structure:
.
├── interface.py
├── pkg
│ ├── __init__.py
│ ├── mod1.py
│ ├── mod2.py
Now, these are my conditions:
mod2 needs to import mod1.
both interface.py and mod2 needs to be run independently as a main script. If you want, think interface as the actual program and mod2 as an internal tester of the package.
So, in Python 2 I would simply do import mod1 inside mod2.py and both python2 mod2.py and python2 interface.py would work as expected.
However, and this is the part I less understand, using Python 3.5.2, if I do import mod1; then I can do python3 mod2.py, but python3 interface.py throws: ImportError: No module named 'mod1' :(
So, apparently, python 3 proposes to use import pkg.mod1 to avoid collisions against built-in modules. Ok, If I use that I can do python3 interface.py; but then I can't python3 mod2.py because: ImportError: No module named 'pkg'
Similarly, If I use relative import:
from . import mod1 then python3 interface.py works; but mod2.py says SystemError: Parent module '' not loaded, cannot perform relative import :( :(
The only "solution", I've found is to go up one folder and do python -m pkg.mod2 and then it works. But do we have to be adding the package prefix pkg to every import to other modules within that package? Even more, to run any scripts inside the package, do I have to remember to go one folder up and use the -m switch? That's the only way to go??
I'm confused. This scenario was pretty straightforward with python 2, but looks awkward in python 3.
UPDATE: I have upload those files with the (referred as "solution" above) working source code here: https://gitlab.com/Akronix/test_python3_packages. Note that I still don't like it, and looks much uglier than the python2 solution.
Related SO questions I've already read:
Python -- import the package in a module that is inside the same package
How to do relative imports in Python?
Absolute import module in same package
Related links:
https://docs.python.org/3.5/tutorial/modules.html
https://www.python.org/dev/peps/pep-0328/
https://www.python.org/dev/peps/pep-0366/
TLDR:
Run your code with python -m pkg.mod2.
Import your code with from . import mod1.
The only "solution", I've found is to go up one folder and do python -m pkg.mod2 and then it works.
Using the -m switch is indeed the "only" solution - it was already the only solution before. The old behaviour simply only ever worked out of sheer luck; it could be broken without even modifying your code.
Going "one folder up" merely adds your package to the search path. Installing your package or modifying the search path works as well. See below for details.
But do we have to be adding the package prefix pkg to every import to other modules within that package?
You must have a reference to your package - otherwise it is ambiguous which module you want. The package reference can be either absolute or relative.
A relative import is usually what you want. It saves writing pkg explicitly, making it easier to refactor and move modules.
# inside mod1.py
# import mod2 - this is wrong! It can pull in an arbitrary mod2 module
# these are correct, they uniquely identify the module
import pkg.mod2
from pkg import mod2
from . import mod2
from .mod2 import foo # if pkg.mod2.foo exists
Note that you can always use <import> as <name> to bind your import to a different name. For example, import pkg.mod2 as mod2 lets you work with just the module name.
Even more, to run any scripts inside the package, do I have to remember to go one folder up and use the -m switch? That's the only way to go??
If your package is properly installed, you can use the -m switch from anywhere. For example, you can always use python3 -m json.tool.
echo '{"json":"obj"}' | python -m json.tool
If your package is not installed (yet), you can set PYTHONPATH to its base directory. This includes your package in the search path, and allows the -m switch to find it properly.
If you are in the executable's directory, you can execute export PYTHONPATH="$(pwd)/.." to quickly mount the package for import.
I'm confused. This scenario was pretty straightforward with python 2, but looks awkward in python 3.
This scenario was basically broken in python 2. While it was straightforward in many cases, it was difficult or outright impossible to fix in any other cases.
The new behaviour is more awkward in the straightforward case, but robust and reliable in any case.
I had similar problem.
I solved it adding
import sys
sys.path.insert(0,".package_name")
into the __init__.py file in the package folder.
I have trouble importing package.
My file structure is like this:
filelib/
__init__.py
converters/
__init__.py
cmp2locus.py
modelmaker/
__init__.py
command_file.py
In module command_file.py I have a class named CommandFile which i want to call in the cmp2locus.py module.
I have tried the following in cmp2locus.py module:
import filelib.modelmaker.command_file
import modelmaker.command_file
from filelib.modelmaker.command_file import CommandFile
All these options return ImportError: No modules named ...
Appreciate any hint on solving this. I do not understand why this import does not work.
To perform these imports you have 3 options, I'll list them in the order I'd prefer. (For all of these options I will be assuming python 3)
Relative imports
Your file structure looks like a proper package file structure so this should work however anyone else trying this option should note that it requires you to be in a package; this won't work for some random script.
You'll also need to run the script doing the importing from outside the package, for example by importing it and running it from there rather than just running the cmp2locus.py script directly
Then you'll need to change your imports to be relative by using ..
So:
import filelib.modelmaker.command_file
becomes
from ..modelmaker import command_file
The .. refers to the parent folder (like the hidden file in file systems).
Also note you have to use the from import syntax because names starting with .. aren't valid identifiers in python. However you can of course import it as whatever you'd like using from import as.
See also the PEP
Absolute imports
If you place your package in site-packages (the directories returned by site.getsitepackages()) you will be able to use the format of imports that you were trying to use in the question. Note that this requires any users of your package to install it there too so this isn't ideal (although they probably would, relying on it is bad).
Modifying the python path
As Meera answered you can also directly modify the python path by using sys.
I dislike this option personally as it feels very 'hacky' but I've been told it can be useful as it gives you precise control of what you can import.
To import from another folder, you have to append that path of the folder to sys.path:
import sys
sys.path.append('path/filelib/modelmaker')
import command_file
Imagine this directory structure:
app/
__init__.py
sub1/
__init__.py
mod1.py
sub2/
__init__.py
mod2.py
I'm coding mod1, and I need to import something from mod2. How should I do it?
I tried from ..sub2 import mod2 but I'm getting an "Attempted relative import in non-package".
I googled around but found only "sys.path manipulation" hacks. Isn't there a clean way?
Edit: all my __init__.py's are currently empty
Edit2: I'm trying to do this because sub2 contains classes that are shared across sub packages (sub1, subX, etc.).
Edit3: The behaviour I'm looking for is the same as described in PEP 366 (thanks John B)
Everyone seems to want to tell you what you should be doing rather than just answering the question.
The problem is that you're running the module as '__main__' by passing the mod1.py as an argument to the interpreter.
From PEP 328:
Relative imports use a module's __name__ attribute to determine that module's position in the package hierarchy. If the module's name does not contain any package information (e.g. it is set to '__main__') then relative imports are resolved as if the module were a top level module, regardless of where the module is actually located on the file system.
In Python 2.6, they're adding the ability to reference modules relative to the main module. PEP 366 describes the change.
Update: According to Nick Coghlan, the recommended alternative is to run the module inside the package using the -m switch.
Here is the solution which works for me:
I do the relative imports as from ..sub2 import mod2
and then, if I want to run mod1.py then I go to the parent directory of app and run the module using the python -m switch as python -m app.sub1.mod1.
The real reason why this problem occurs with relative imports, is that relative imports works by taking the __name__ property of the module. If the module is being directly run, then __name__ is set to __main__ and it doesn't contain any information about package structure. And, thats why python complains about the relative import in non-package error.
So, by using the -m switch you provide the package structure information to python, through which it can resolve the relative imports successfully.
I have encountered this problem many times while doing relative imports. And, after reading all the previous answers, I was still not able to figure out how to solve it, in a clean way, without needing to put boilerplate code in all files. (Though some of the comments were really helpful, thanks to #ncoghlan and #XiongChiamiov)
Hope this helps someone who is fighting with relative imports problem, because going through PEP is really not fun.
main.py
setup.py
app/ ->
__init__.py
package_a/ ->
__init__.py
module_a.py
package_b/ ->
__init__.py
module_b.py
You run python main.py.
main.py does: import app.package_a.module_a
module_a.py does import app.package_b.module_b
Alternatively 2 or 3 could use: from app.package_a import module_a
That will work as long as you have app in your PYTHONPATH. main.py could be anywhere then.
So you write a setup.py to copy (install) the whole app package and subpackages to the target system's python folders, and main.py to target system's script folders.
"Guido views running scripts within a package as an anti-pattern" (rejected
PEP-3122)
I have spent so much time trying to find a solution, reading related posts here on Stack Overflow and saying to myself "there must be a better way!". Looks like there is not.
This is solved 100%:
app/
main.py
settings/
local_setings.py
Import settings/local_setting.py in app/main.py:
main.py:
import sys
sys.path.insert(0, "../settings")
try:
from local_settings import *
except ImportError:
print('No Import')
explanation of nosklo's answer with examples
note: all __init__.py files are empty.
main.py
app/ ->
__init__.py
package_a/ ->
__init__.py
fun_a.py
package_b/ ->
__init__.py
fun_b.py
app/package_a/fun_a.py
def print_a():
print 'This is a function in dir package_a'
app/package_b/fun_b.py
from app.package_a.fun_a import print_a
def print_b():
print 'This is a function in dir package_b'
print 'going to call a function in dir package_a'
print '-'*30
print_a()
main.py
from app.package_b import fun_b
fun_b.print_b()
if you run $ python main.py it returns:
This is a function in dir package_b
going to call a function in dir package_a
------------------------------
This is a function in dir package_a
main.py does: from app.package_b import fun_b
fun_b.py does from app.package_a.fun_a import print_a
so file in folder package_b used file in folder package_a, which is what you want. Right??
def import_path(fullpath):
"""
Import a file with full path specification. Allows one to
import from anywhere, something __import__ does not do.
"""
path, filename = os.path.split(fullpath)
filename, ext = os.path.splitext(filename)
sys.path.append(path)
module = __import__(filename)
reload(module) # Might be out of date
del sys.path[-1]
return module
I'm using this snippet to import modules from paths, hope that helps
This is unfortunately a sys.path hack, but it works quite well.
I encountered this problem with another layer: I already had a module of the specified name, but it was the wrong module.
what I wanted to do was the following (the module I was working from was module3):
mymodule\
__init__.py
mymodule1\
__init__.py
mymodule1_1
mymodule2\
__init__.py
mymodule2_1
import mymodule.mymodule1.mymodule1_1
Note that I have already installed mymodule, but in my installation I do not have "mymodule1"
and I would get an ImportError because it was trying to import from my installed modules.
I tried to do a sys.path.append, and that didn't work. What did work was a sys.path.insert
if __name__ == '__main__':
sys.path.insert(0, '../..')
So kind of a hack, but got it all to work!
So keep in mind, if you want your decision to override other paths then you need to use sys.path.insert(0, pathname) to get it to work! This was a very frustrating sticking point for me, allot of people say to use the "append" function to sys.path, but that doesn't work if you already have a module defined (I find it very strange behavior)
Let me just put this here for my own reference. I know that it is not good Python code, but I needed a script for a project I was working on and I wanted to put the script in a scripts directory.
import os.path
import sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
As #EvgeniSergeev says in the comments to the OP, you can import code from a .py file at an arbitrary location with:
import imp
foo = imp.load_source('module.name', '/path/to/file.py')
foo.MyClass()
This is taken from this SO answer.
Take a look at http://docs.python.org/whatsnew/2.5.html#pep-328-absolute-and-relative-imports. You could do
from .mod1 import stuff
From Python doc,
In Python 2.5, you can switch import‘s behaviour to absolute imports using a from __future__ import absolute_import directive. This absolute- import behaviour will become the default in a future version (probably Python 2.7). Once absolute imports are the default, import string will always find the standard library’s version. It’s suggested that users should begin using absolute imports as much as possible, so it’s preferable to begin writing from pkg import string in your code
I found it's more easy to set "PYTHONPATH" enviroment variable to the top folder:
bash$ export PYTHONPATH=/PATH/TO/APP
then:
import sub1.func1
#...more import
of course, PYTHONPATH is "global", but it didn't raise trouble for me yet.
On top of what John B said, it seems like setting the __package__ variable should help, instead of changing __main__ which could screw up other things. But as far as I could test, it doesn't completely work as it should.
I have the same problem and neither PEP 328 or 366 solve the problem completely, as both, by the end of the day, need the head of the package to be included in sys.path, as far as I could understand.
I should also mention that I did not find how to format the string that should go into those variables. Is it "package_head.subfolder.module_name" or what?
You have to append the module’s path to PYTHONPATH:
export PYTHONPATH="${PYTHONPATH}:/path/to/your/module/"
A hacky way to do it is to append the current directory to the PATH at runtime as follows:
import pathlib
import sys
sys.path.append(pathlib.Path(__file__).parent.resolve())
import file_to_import # the actual intended import
In contrast to another solution for this question this uses pathlib instead of os.path.
This method queries and auto populates the path:
import os
import inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
os.sys.path.insert(1, parentdir)
# print("currentdir = ", currentdir)
# print("parentdir=", parentdir)
What a debate!
Relative newcomer to python (but years of programming experience, and dislike of perl). Relative lay-person when it comes to the dark art of Apache setup, but I know what I (think I) need to get my little experimental projects working at home.
Here is my summary of what the situ seems to be.
If I use the -m 'module' approach, I need to:-
dot it all together;
run it from a parent folder;
lose the '.py';
create an empty (!) __init__.py file in every sub-folder.
How does that work in a cgi environment, where I have aliased my scripts directory, and want to run a script directly as /dirAlias/cgi_script.py??
Why is amending sys.path a hack? The python docs page states: "A program is free to modify this list for its own purposes." If it works, it works, right? The bean counters in Accounts don't care how it works.
I just want to go up one level and down into a 'modules' dir:-
.../py
/cgi
/build
/modules
so my 'modules' can be imported from either the cgi world or the server world.
I've tried the -m/modules approach but I think I prefer the following (and am not confused how to run it in cgi-space):-
Create XX_pathsetup.py in the /path/to/python/Lib dir (or any other dir in the default sys.path list). 'XX' is some identifier that declares an intent to setup my path according to the rules in the file.
In any script that wants to be able to import from the 'modules' dir in above directory config, simply import XX_pathsetup.py.
And here's my really simple XX_pathsetup.py:
import sys, os
pypath = sys.path[0].rsplit(os.sep,1)[0]
sys.path.insert( 0, pypath+os.sep+'modules' )
Not a 'hack', IMHO. 1 small file to put in the python 'Lib' dir, one import statement which declares intent to modify the path search order.
I can't figure out how to import modules from sibling directories in Python 3 using absolute imports.
modify the sys.path.
turn the directory into a pip installable package via __init__.py and setup.py.
For option 1. I figured out how to import modules from sibling directories by modifying the sys.path, but this method seems a little hackey to me. Also, I've read that it is not preferred. Why? Is there something inherently wrong or dangerous about modifying the sys.path?
For option 2. What exactly do I need to do make my package pip installable? I've alreay created my __init__.py file, but it seems that I need to create and configure a setup.py script to prepare my package for distribution? I'm still in the development mode, so is this really the best/pythonic method? If it is, then do I just type python setup.py install into my terminal after creating the setup.py?
Edit: I'm now trying to figure this out using absolute imports as python 3 does support relative imports.
From what I've read, Python 3 does not support relative imports
It does.
To import myproject/foo/__init__.py from myproject/bar/baz.py, you can use this:
from .. import foo
Or if you want to import an object/module in foo:
from ..foo import object
This requires myproject to be a package, so myproject/__init__.py has to exist.
I have the following directory structure:
some-tools-dir/
base_utils.py
other_utils.py
some-tool.py
some-other-tool.py
some-other-tools-dir/
basetools -> symlink to ../some-tools-dir
yet-another-tool.py
In other_utils.py, I have:
import base_utils
Now, in yet-another-tool.py, I want to do:
import basetools.other_utils
That doesn't work, because Python does not recognize basetools as a Python package.
So I add an empty basetools/__init__.py.
Now, in other_utils, I get the exception:
import base_utils
ImportError: No module named base_utils
So I change that line to:
from . import base_utils
And yet-another-tool.py works now.
However, some-tool.py does not work anymore. It imports other_utils, and there I get the exception:
from . import base_utils
ValueError: Attempted relative import in non-package
Now, I can add this hack/workaround to some-tools-dir/*-tool.py:
import os, sys
__package__ = os.path.basename(os.path.dirname(os.path.abspath(__file__)))
sys.path += [os.path.dirname(os.path.dirname(os.path.abspath(__file__)))]
__import__(__package__)
And in addition, make all local imports relative in those files.
That solves the problem, I guess. However, it looks somewhat very ugly and I have to modify sys.path. I tried several variations of this hack, however, I want to support multiple Python versions if possible, so using the module importlib becomes complicated, esp. because I have Python 3.2, and I don't like using module imp because it's deprecated. Also, it only seems to become more complicated.
Is there something I'm missing? This all looks ugly and too complicated for a use-case which doesn't seem to be too uncommon (for me). Is there a cleaner/simpler version of my hack?
A restriction I'm willing to make is to only support Python >=3.2, if that simplifies anything.
(Note that this answer was built by piecing together information from this answer and this question, so go up-vote them if you like it)
This looks a bit less hacky, and at least works with Python 2.7+:
if __name__ == "__main__" and __package__ is None:
import sys, os.path as path
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
from some_tools_dir import other_utils
I think the main reason you're finding this difficult is because it's actually unusual to have executable scripts inside of a python package. Guido van Rossum actualy calls it an "antipattern". Normally your executable lives above the root directory of the package, and then can simply use:
from some_tools_dir import other_utils
Without any fuss.
Or, if you want to execute a script that lives in the package, you actually call it as part of the package (again, from the parent dir of the package):
python -m some_tools_dir.other_utils
Are you be able to add the top root path into PYTHONPATH ?
If so, you can then add
__init__.py
file into some-tools-dir (and/or some-other-tools-dir)
Then from other_utils.py you do
from some-tools-dir import base_utils
And in yet-another-tool.py you do
from some-tools-dir import other_utils
You can then remove the symlink, and have proper namespacing.