I was wondering if anyone had any suggestions for lazy loading imports in an init file? I currently have the following folder structure:
/mypackage
__init__.py
/core
__init__.py
mymodule.py
mymodule2.py
The init.py file in the core folder with the following imports:
from mymodule import MyModule
from mymodule2 import MyModule2
This way I can just do:
from mypackage.core import MyModule, MyModule2
However, in the package init.py file, I have another import:
from core.exc import MyModuleException
This has the effect that whenever I import my package in python, MyModule and MyModule2 get imported by default because the core init.py file has already been run.
What I want to do, is only import these modules when the following code is run and not before:
from mypackage.core import MyModule, MyModule2
Any ideas?
Many thanks.
Unless I'm mistaking your intentions, this is actually possible but requires some magic.
Basically, subclass types.ModuleType and override __getattr__ to import on demand.
Check out the Werkzeug init.py for an example.
You can't. Remember that when python imports it executes the code in the module. The module itself doesn't know how it is imported hence it cannot know whether it has to import MyModule(2) or not.
You have to choose: allow from mypackage.core import A, B and from core.exc import E does the non-needed imports (x)or do not import A and B in core/__init__.py, hence not allowing from mypackage.core import A, B.
Note: Personally I would not import MyModule(2) in core/__init__.py, but I'd add an all.py module that does this, so the user can do from mypackage.core.all import A, B
and still have from mypackage.core.exc import TheException not loading the unnecessary classes.
(Actually: the all module could even modify mypackage.core and add the classes to it, so that following imports of the kind from mypackage.core import MyModule, MyModule2 work, but I think this would be quite obscure and should be avoided).
If your modules structure is like:
/mypackage
__init__.py
/core
__init__.py
MyModule.py
MyModule2.py
or:
/mypackage
__init__.py
/core
__init__.py
/MyModule
__init__.py
/MyModule2
__init__.py
then feel free to use
from mypackage.core import MyModule, MyModule2
without importing them in __init__.py under mypackage/core
Not sure if it applies here but in general lazy loading of modules can be done using the Importing package.
Works like this:
from peak.util.imports import lazyModule
my_module = lazyModule('my_module')
Now my module is only really imported when you use it the first time.
You may use follow code in __init__ in module:
import apipkg
apipkg.initpkg(__name__, {
'org': {
'Class1': "secure._mypkg:Class1",
'Class2': "secure._mypkg2:Class2",
}
})
I realize that this question was posted a very long time ago and since then there has been some helpful updates to solve lazy loading submodules. So for anyone else looking for how to solve this we have great options available now.
Specifically PEP 562
Here is an excerpt from that article:
Another widespread use case for getattr would be lazy submodule imports. > Consider a simple example:
# lib/__init__.py
import importlib
__all__ = ['submod', ...]
def __getattr__(name):
if name in __all__:
return importlib.import_module("." + name, __name__)
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
# lib/submod.py
print("Submodule loaded")
class HeavyClass:
...
# main.py
import lib
lib.submod.HeavyClass # prints "Submodule loaded"
Related
I am working with the following directory in Python3.8:
package/
__init__.py
/folder1
__init__.py
file1.py
/folder2
__init__.py
file2.py
/folder3
__init__.py
file3.py
I would like to import a function from file3 into file2. What is the easiest way to do it? I would also like to avoid manually appending to PYTHONPATH, to make this as smooth as possible for the next person pulling the project.
So far tried relative imports, which didn't work, possibly because I did not specify something in the init, which is currently empty. The error I am getting using:
from ..package.folder3.file3 import function_name
is
ImportError: attempted relative import with no known parent package
Thanks for all help!
The answer to your question is pretty simple, you need to add your package path to the system path.
Here is a complete example:
In file3.py, let's create a simple function
def func():
print("Hello from file3")
In file2.py, we can import func function like so:
import os
import sys
sys.path.append(os.path.abspath('../../..'))
# import func now from file3
from package.folder3.file3 import func
func() #should return "Hello from file3"
Hopefully, this answers your question!
I have been wondering for a long time how to appropriately import in Python, in relation to the way I'm going to explain now:
Imagine I've got two modules built by myself, modulea and moduleb, and have a last one which uses both the modulea and b, so modulec.
Furthermore, modulea uses moduleb.
The modules are all one-file, which form part of a big module which has got its own __init__.py file.
+ My project
-
-> modulea.py
-> moduleb.py
-> modulec.py
-> __init__.py # Here I import modulea, b, c.
-> main_module.py # This is were I do all the hardcore.
I've created a file for importing all the modules I need, instead of importing them one by one in each script, and to know if I have already imported it, I would have to check file for file if I have already imported that or not.
I could simply declare, as you have been saying below in the comment box, the required modules in each module, but I think that's not practical and it's easier to import them all only in one file.
Tell me if this is a good practice or not.
If I import
import modulea
import moduleb
import modulec
It won't work (ImportError) because modulea does not implicitly import moduleb inside the code, and it is imported before than the module it requires (b)
If I did
import moduleb
import modulea
import modulec
I guess they will work because since the runnning script already imports moduleb before a, it should also be accessible to a and not only to the running script; and the same I guess it happens with modulec.
Supposing this is true, which I think indeed, will it work if the import were done in only one line? Are the modules imported and included into the script one by one or are they rather imported altogether and them enabled for use, after the statement finishes (the one-line import call)?
import moduleb, modulea, modulec
And if it does, does it shorten too much the processing time of the code, or is it basically the same, despite the aesthetic matter?
You are overthinking way too much.
Each module must be "self contained" in the sense that itself satisfies its dependencies. If modulea needs moduleb, you do
# modulea.py
import moduleb
Then moduleb has to import its dependencies, if any:
# moduleb.py
import json # for instance
And finally modulec needs modulea and moduleb, you just import them:
# modulec.py
import modulea
import moduleb # <-- actually this import does not reload the moduleb, since it was already loaded in modulea, and it was cached.
There is nothing more about this.
And if you just need a function from a module, you can just import it into the namespace like this.
from moduleb import my_cool_function
Just as style convention, you should do each import in a new line:
import json
from moduleb import my_cool_function
from moduleb import another_not_so_cool_function
According to the python style guide you should use an import statement for each individual module. It will also tell you everything else you need to know about the conventions surrounding importing modules.
import moduleb
import modulea
import modulec
Also, as some of the commenters have indicated, you don't seem to be importing these modules properly within each script, thus creating errors here in this script.
You can put all of your imports into one place if you'd like.
myproject/internal.py
from moduleb import *
from modulea import *
from modulec import *
Now, your other modules can just import it
myproject/modulec.py
from internal import *
Although its usually preferable to import everything in the module that uses it, there are conditions where that is not desirable. I have a project with many small scripts that test product functionality. It would be a bother to have lots of imports in each of them.
Currently have the following file hierarchy:
\package
__init__.py
run_everything.py
\subpackage
__init__.py
work.py
work1.py
work2.py
\test
__init__.py
test_work.py
test_work1.py
My first question is regarding relative imports. Suppose in \subpackage\work.py I have a function called custom_function(), and I would like to test that function in test_work.py. For some reason I can not figure out how to make this import from one module to another. Trying from .. subpackage.work1 import custom_function() does not seem to work, and yields the error Attempted relative import in non-package Is there any way to resolve this?
2)
I would like to run all test files from run_everything.py with one function, would adding a suite() function in each test_work*.py file, which adds each unit_testing class to suite.addTest(unittest.makeSuite(TestClass)), and finally importing them into the top-level run_everything.py be the most conventional way in Python2.7?
Here is a hack*
Insert the path's to "subpackage" and "test" to your python path in run_everything using:
import sys
sys.path.insert(0, '/path/to/package/subpackage')
sys.path.insert(0, '/path/to/package/test')
And then, you can import all your files using vanilla imports in run_everything:
import work, work1, work2
import test_work, test_work1
*This won't permanently affect your PYTHONPATH.
I have a Django project located at /var/django/project/ where /var/django/ is in the PATH
within that project I have:
___init__.py
manage.py
utils/
__init__.py
tools.py
utils/__init__.py contains a function named get_preview
utils/tools.py contains a function named get_related
How can utils/__init__.py import get_related from utils/tools.py?
How can utils/tools.py import get_preview from utils/__init_.py?
I have tried relative imports as well as static imports but seem to get an error in tools.py when I try to from project.utils import get_preview
Yeah, this is bad structure. You gotta watch out here with creating a circular import between the two files.
About circular imports.
You can't (and shouldn't). You are structuring your code very poorly if files in your module are referencing code in the __init__.py associated with it. Either move both functions into __init__.py or both of them out of __init__.py or put them into separate modules. Those are your only options.
You can do it, you just need to make one of the imports happen at runtime to avoid the circular import.
For example, __init__.py:
from project.utils.tools import get_related
def get_preview():
# ...
and tools.py:
def get_related():
from project.utils import get_preview
# ...
get_preview()
I have a directory structure that looks like this:
project/
__init__.py
foo/
__init.py__
first.py
second.py
third.py
plum.py
In project/foo/__init__.py I import classes from first.py, second.py and third.py and put them in __all__.
There's a class in first.py named WonderfulThing which I'd like to use in second.py, and want to import by importing * from foo. (It's outside of the scope of this question why I'd like to do so, assume I have a good reason.)
In second.py I've tried from .foo import *, from foo import * and from . import * and in none of these cases is WonderfulThing imported. I also tried from ..foo import *, which raises an error "Attempted relative import beyond toplevel package".
I've read the docs and the PEP, and I can't work out how to make this work. Any assistance would be appreciated.
Clarification/Edit: It seems like I may have been misunderstanding the way __all__ works in packages. I was using it the same as in modules,
from .first import WonderfulThing
__all__ = [ "WonderfulThing" ]
but looking at the docs again it seems to suggest that __all__ may only be used in packages to specify the names of modules to be imported by default; there doesn't seem to be any way to include anything that's not a module.
Is this correct?
A non-wildcard import failed (cannot import name WonderfulThing). Trying from . import foo failed, but import foo works. Unfortunately, dir(foo) shows nothing.
Edit: I did misunderstand the question: No __all__ is not restricted to just modules.
One question is why you want to do a relative import. There is nothing wrong with doing from project.foo import *, here. Secondly, the __all__ restriction on foo won't prevent you from doing from project.foo.first import WonderfulThing, or just from .first import WonderfulThing, which still will be the best way.
And if you really want to import a a lot of things, it's probably best to do from project import foo, and then use the things with foo.WonderfulThing instead for doing an import * and then using WonderfulThing directly.
However to answer your direct question, to import from the __init__ file in second.py you do this:
from . import WonderfulThing
or
from . import *