EDIT: A circular dependency issue was the reason why from .. import auth didn't work, see my own answer below. Cheers to all.
I'm dabbling with relative imports in Python while reorganizing some code and what should have taken 10 minutes is now taking untold amounts of time. Here's my file structure:
project/
/__init__.py
/application.py
/lib/
/__init__.py
/auth.py
/utils.py
/views/
/__init__.py
/login_view.py
/test.py
In login_view.py I have:
from . import auth
which yields: ImportError: cannot import name auth
However this works:
from ..auth import untoken, req_auth_or_error
Which I don't understand: auth is only one level above login_view, why the .. and not .?
This also fails with the same error:
from .. import auth
In test.py I tried the following:
import types
from .. import *
def imports():
for name, val in globals().items():
if isinstance(val, types.ModuleType):
print 'imported: ', val.__name__
imports()
which results in:
imported: lib.utils
imported: lib.views
imported: types
... I'm at a loss as to why lib.auth is not imported. What am I doing wrong?
I'm running the code as such (the abs. path is /scratch/project/):
$ cd project/
$ ./application.py
In Python a single dot in import represents the current package, NOT the current module.
Then every additional dot represents a new level of "parentness" (ok this word does not exist but I think it expresses well what I mean).
So the behaviour you notice in your first case is just normal.
For your second case, I to reproduced your hierarchy with those contents:
application.py
lib.views import login_view
print("oh yeah")
login_view.py
lib.views import login_view
print("oh my god")
... and running application.py from the project directory does not cause any error. I get the following output:
$ python application.py
oh my god
oh yeah
However, adding a from lib.views import test (using the exact same test.py content you provided) in the application.py gives me the following output showing that some lib modules are not loaded:
$ python application.py
oh my god
imported: types
imported: lib.views
imported: lib.auth
oh yeah
The lib.utils module is missing :/
What I think is that since from .. import * names nothing: no package, no module, no function or variable or anything, the import statement simply imports nothing. TH displayed modules/packages are only the one which have been loaded earlier.
That said, there is a way to force a "correct" behaviour by using the __all__ magic variable, which defines which modules are loaded when using the splat import, in the lib/__init__.py file:
__all__ = ['auth', 'views', 'utils']
After doing this, running again the application.py file shows the following:
$ python application.py
oh my god
imported: types
imported: lib.auth
imported: lib.views
imported: lib.utils
oh yeah
In the first case, from . import auth the . means the package views. That is why Python can't find it.
But, to avoid those problems is recommended that you use absolute import instead of relative imports.
What if you went up 1 more directory when importing?
from ... import auth
The problem was a circular dependency one and not one of relative imports.
Indeed, .. is needed to load a module from the parent directory.
However somewhere in auth.py I had an import views.login_view. I believe that the from .. import auth in login_view.py tried to import the auth module as it was being loaded itself. This is probably why from ..auth import untoken works because those functions had already been loaded.
Related
The project has the same structure as in the picture: I'm trying to import from "mod.py " in "index.py "
from .. import mod
However, it gives the error: "ImportError: attempted relative import with no known parent package" If you use this option:
from pack1 import mod
Then error: "ModuleNotFoundError error: there is no module named 'pack1'"
enter image description here
PROJECT/
pack1/
__init__.py
mod.py
pack2/
__init__.py
index.py
What is the problem?
This is a recurring question on StackOverflow. And much of the confusion (in my opinion) comes from how Python interprets the files and folders it sees is based on where Python is run from. First, some terminology:
module: a file containing Python code.
package: a folder containing files with Python code and other folders.
When you start Python in a directory (folder), it doesn't "know" what the namespace of that directory should be. I.e., if you are working in Z:\path\to_my\project\ when you start Python:
it does NOT consider project to be a package.
any .py files you want to import from will be in their own namespace as modules.
any folders you want to import from will also be in their own namespace as packages.
What about __init__.py? Since version 3.3, Python has implicit namespace packages, which allows importing without needing to create an empty __init__.py file.
Consider #2: if you have two files: first.py and second.py:
path/
to_my/
project/
>>Python is running here<<
first.py
second.py
with these contents:
# first.py
first_var = 'hello'
# second.py
from .first import first_var
second_var = first_var + ' world'
if you try to import like this:
>>> import second
Python basically does the following:
"ok, I see second.py"
"Reading that in as a module, chief!"
"Ok, it wants to import .first
"The . means get the package (folder) that contains first.py"
"Wait, I don't have a parent package for first.py!"
"Better raise an error."
The same rules apply for #3 as well. If we add a few packages to the project like this:
path/
to_my/
project/
>>Python is running here<<
first.py
second.py
pack1/
mod.py
other_mod.py
pack2/
index.py
with the following contents:
# pack1/mod.py
mod_var = 1234
# pack1/other_mod.py
from .mod import mod_var
other_var = mod_var * 10
# pack2/index.py
from ..pack1 import mod
and when you try to import like this:
>>> from pack2 import index.py
The import in pack2/index.py is going to fail for the same reason second.py, Python will work its way up the import chain of dots like this:
"Reading in in index.py as a module."
"Looks like it wants to import mod from ..pack1.
"Ok, . is the pack2 parent package namespace of index.py, found that."
"So, .. is the parent package of pack2."
"But, I don't have a parent package for pack2!"
"Better raise an error."
How do we make it work? Two thing.
First, move where Python is running up one level so that all of the .py files and subfolders are considered to be part of the same package namespace, which allows the file to reference each other using relative references.
path/
to_my/
>>Python is running here now<<
project/
first.py
second.py
pack1/
mod.py
other_mod.py
pack2/
index.py
So now Python sees project as a package namespace, and all of the files within can use relative references up to that level.
This changes how you import when you are in the Python interpreter:
>>> from project.pack2 import index.py
Second, you make explicit references instead of relative references. That can make the import statements really long, but if you have several top-level modules that need to pull from one another, this is how you can do it. This is useful when you are defining your functions in one file and writing your script in another.
# first.py
first_var = 'hello'
# second.py
from first import first_var # we dropped the dot
second_var = first_var + ' world'
I hope this helps clear up some of the confusion about relative imports.
I hope this is not a duplicate, I couldn't find any other answer.
Going straight to the point, my problem is as follows.
I have a project in Django where django-apps use external custom modules.
This is the structure:
Project_dir/
- core/
- module_1.py
- module_2.py
- django_project/
- __init__.py
- settings.py
- urls.py
- wsgi.py
- django_app_A/
- views.py
- manage.py
The problem is that I need to import some classes and methods of moudule_2 in module_1, and I would do so by simply, in module_1,
from module_2 import foo
When I run module_1 for testing, everything works fine. Nonetheless, I need to import module_1 in django_app_A/views.py, and I would do so by
from core.module_1 import Bar
Here's the problem: if I have another relative import in module_1, as I have, I will get a
ModuleNotFoundError: No module named 'module_2'
UNLESS I use in module_1 the syntax
from .module_2 import foo
By doing so, the Django app will work fine and page will properly load, but at the same time I "break" the module_1, as I won't be able to run it stand-alone anymore, as I will get a
ModuleNotFoundError: No module named '__main__.module_2'
I have no idea how to fix this conflict and make both import syntax work at the same time.
Any clues? Am I missing something?
Thanks in advance
You should use absolute imports as much as you can.
from core.module_2 import foo
I can't be sure, but it sounds like a circular import problem to me.
Do you need the import to be on the "main" level? If you import module 2 inside of a class or function, simply write
def function_in_question():
import module_1
return module_1.whatever()
Another thing to look for: are you using it in a way that is actually circular? A function in module_2 using a function in module_1 that uses the function in module_2?
I have sub-directories inside my django app folder, and on each I was trying to call a module. The issue that I am having is that I am able to import a module by using * but not by name which produces an error "Exception Value: cannot import name [my module]"
from foo import Bar # throws error
from foo import * # works
I dont know if I am missing anything on my settings.py but definitely I have included the app directory on my INSTALLED_APPS and also I have init.py on each directory. I also tried to check if my app folder is included on python paths and it was included.
Any help will be appreciated. THanks in advance
I expect you are thinking in terms of Java. In Python, you import things by module, not class name. So if a directory foo contains a file bar.py which defines a class Bar, you must do from foo.bar import Bar, not from foo import Bar.
After looking over my directories I found out that I have a file that has the same name as my app/Foo. after removing this it started working.
Foo/
- Bar.py
Process/
- Foo.py # deleted this
- views.py
from Foo import Bar #Foo.py was overriding the app/Foo that I was trying to call
Thanks for all your response!
if this is in django, try doing
from . import form
from . import models
from . import views
This should solve the issue.
Apart from that, fist make sure you have a init.py file in the directory.
I was wondering if anyone had any suggestions for lazy loading imports in an init file? I currently have the following folder structure:
/mypackage
__init__.py
/core
__init__.py
mymodule.py
mymodule2.py
The init.py file in the core folder with the following imports:
from mymodule import MyModule
from mymodule2 import MyModule2
This way I can just do:
from mypackage.core import MyModule, MyModule2
However, in the package init.py file, I have another import:
from core.exc import MyModuleException
This has the effect that whenever I import my package in python, MyModule and MyModule2 get imported by default because the core init.py file has already been run.
What I want to do, is only import these modules when the following code is run and not before:
from mypackage.core import MyModule, MyModule2
Any ideas?
Many thanks.
Unless I'm mistaking your intentions, this is actually possible but requires some magic.
Basically, subclass types.ModuleType and override __getattr__ to import on demand.
Check out the Werkzeug init.py for an example.
You can't. Remember that when python imports it executes the code in the module. The module itself doesn't know how it is imported hence it cannot know whether it has to import MyModule(2) or not.
You have to choose: allow from mypackage.core import A, B and from core.exc import E does the non-needed imports (x)or do not import A and B in core/__init__.py, hence not allowing from mypackage.core import A, B.
Note: Personally I would not import MyModule(2) in core/__init__.py, but I'd add an all.py module that does this, so the user can do from mypackage.core.all import A, B
and still have from mypackage.core.exc import TheException not loading the unnecessary classes.
(Actually: the all module could even modify mypackage.core and add the classes to it, so that following imports of the kind from mypackage.core import MyModule, MyModule2 work, but I think this would be quite obscure and should be avoided).
If your modules structure is like:
/mypackage
__init__.py
/core
__init__.py
MyModule.py
MyModule2.py
or:
/mypackage
__init__.py
/core
__init__.py
/MyModule
__init__.py
/MyModule2
__init__.py
then feel free to use
from mypackage.core import MyModule, MyModule2
without importing them in __init__.py under mypackage/core
Not sure if it applies here but in general lazy loading of modules can be done using the Importing package.
Works like this:
from peak.util.imports import lazyModule
my_module = lazyModule('my_module')
Now my module is only really imported when you use it the first time.
You may use follow code in __init__ in module:
import apipkg
apipkg.initpkg(__name__, {
'org': {
'Class1': "secure._mypkg:Class1",
'Class2': "secure._mypkg2:Class2",
}
})
I realize that this question was posted a very long time ago and since then there has been some helpful updates to solve lazy loading submodules. So for anyone else looking for how to solve this we have great options available now.
Specifically PEP 562
Here is an excerpt from that article:
Another widespread use case for getattr would be lazy submodule imports. > Consider a simple example:
# lib/__init__.py
import importlib
__all__ = ['submod', ...]
def __getattr__(name):
if name in __all__:
return importlib.import_module("." + name, __name__)
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
# lib/submod.py
print("Submodule loaded")
class HeavyClass:
...
# main.py
import lib
lib.submod.HeavyClass # prints "Submodule loaded"
Currently have the following file hierarchy:
\package
__init__.py
run_everything.py
\subpackage
__init__.py
work.py
work1.py
work2.py
\test
__init__.py
test_work.py
test_work1.py
My first question is regarding relative imports. Suppose in \subpackage\work.py I have a function called custom_function(), and I would like to test that function in test_work.py. For some reason I can not figure out how to make this import from one module to another. Trying from .. subpackage.work1 import custom_function() does not seem to work, and yields the error Attempted relative import in non-package Is there any way to resolve this?
2)
I would like to run all test files from run_everything.py with one function, would adding a suite() function in each test_work*.py file, which adds each unit_testing class to suite.addTest(unittest.makeSuite(TestClass)), and finally importing them into the top-level run_everything.py be the most conventional way in Python2.7?
Here is a hack*
Insert the path's to "subpackage" and "test" to your python path in run_everything using:
import sys
sys.path.insert(0, '/path/to/package/subpackage')
sys.path.insert(0, '/path/to/package/test')
And then, you can import all your files using vanilla imports in run_everything:
import work, work1, work2
import test_work, test_work1
*This won't permanently affect your PYTHONPATH.