I have the following package structure
package
__init__.py
sub1
__init__.py
foo.py # Contains class Foo
sub2
__init__.py
bar.py # Contains class Bar
I want to be able to just import package and have package.Foo and package.Bar, i.e. I want to have the subpackages be transparent to users.
The catch is that importing sub2 takes a long time, and many users don't care at all about the stuff in sub2 and only want the stuff in sub1. Thus I want users to be able to say import package.sub1 or from package import sub1 to just import sub1 and skip the import of sub2.
I know I can achieve the first part by having package/__init__.py contain
from .sub1 import *
from .sub2 import *
and having package/sub1/__init__.py be from .foo import Foo and similarly for sub2. However, this will always import sub1 and sub2 even if the user tries to import only package.sub1.
Correspondingly, I can achieve the second part by having package/__init__.py be empty and using the same sub1/__init__.py as above. However, then just saying import package doesn't load sub1 or sub2, so users would have to explicitly load them and then refer to package.sub1.Foo.
Ideally a solution would work both in 2.7.10 and 3.5.0, but I'll accept one or the other if both isn't possible.
The LazyLoader class is provided for exactly this kind of situation: postponing loading of the module when it is actually used, instead of at the point of importing it.
To build a lazy loader you can follow the example in the documentation:
suffixes = importlib.machinery.SOURCE_SUFFIXES
loader = importlib.machinery.SourceFileLoader
lazy_loader = importlib.util.LazyLoader.factory(loader)
finder = importlib.machinery.FileFinder(path, [(lazy_loader, suffixes)])
then you can use finder.find_spec to obtain the spec of a module and pass the result to Loader.create_module to load it.
This is a bit cumbersome to do manually for just one module.
Note that searching for "lazy import python" you'll find quite a few solutions that have different pro and cons, some of which run in python2.x. However the LazyLoader class above is the official way of doing it in python3.5+
you could add your shortcuts to the __init__.py of your module:
package/__init__.py
__all__ = [
... add everything you want to be listed for this module
'Foo',
'Bar',
...
]
from package.sub1.foo import Foo
from package.sub2.bar import Bar
now you should be able to call:
from package import Bar
Related
I have the following structure
my_scripts
group_A
group_A1
main.py
dbFunctions.py
group_A2
main.py
dbFunctions.py
group_B
scripting_Im_running_things_from.py
dbFunctions.py
I want to load one function from each of the main.pys, and am trying to do it with sys.path.append. But because the folder I am running my main script from (group_B) has dbFunctions.py as well, none of the other two models' functions (ie. group_A1\main.py and group_A2\main.py) can utilise functions from their respective dbFunctions.py modules so I am getting an import error:
ImportError: cannot import name 'my_function_in_groupA1_dbFunctions' from 'dbFunctions' (C:\my_scripts\group_B\dbFunctions.py)
(I also tried renaming group_B\dbFunctions.py to group_B\_dbFunctions.py and adding the group_A1 and group_A2
to sys.path does allow me to import functions from whichever of group_A1's and group_A2's main.py
I choose to import from first, but then for the second one it is impossible to make it look at the
second folder; because it has loaded one function from a given main.py, seemingly no other main.py > can be considered.)
Is it preferebale to add init.pys in all folders or can this be done by using importlib ?
Apologies for not creating a re-producible example, I cannot do it I believe for this question.
Use relative imports. For example:
# ../my_scripts/group_A/group_A1/main.py
# Import foo from ../my_scripts/group_A/group_A1/dbFunctions.py
from .dbFunctions import foo
# Import bar from ../my_scripts/group_A/group_A2/dbFunctions.py
from ..group_A2.dbFunctions import bar
# Import baz from ../my_scripts/group_B/dbFunctions.py
from ...dbFunctions import baz
Assume the following structure...
root/test.py
root/myapp/__init__.py
root/myapp/myapp.py # contains My_App_Class()
root/myapp/SomeObject.py # contains My_Obj_Class()
If my __init__.py contains the following structure:
from myapp import *
import SomeObject
__all__ = ['SomeObject']
I want to be able to call myapp and have the routines pre-extracted from myapp/myapp.py so that I can do the following in test.py:
import myapp
if 'My_App_Class' in myapp.__dict__.keys():
print 'success'
else:
print 'fail'
if 'My_Obj_Class' in myapp.SomeObject.__dict__.keys():
print 'success'
else:
print 'fail'
so that I can effectively collapse from myapp.myapp import * into from myapp import *
In [1]: %run test.py
fail
success
The __all__ attribute in your __init__.py file is what is preventing other names (imported from myapp/myapp.py) from being visible when using the myapp package.
Just don't use the all - and maybe, to avoid ambiguity, change the import written as:
from myapp import *
to
from .myapp import *
In other words, your __init__.py file should be simply:
from .myapp import *
import SomeObject
and no other line.
Your test.py on the other hand if further incorrect - if it starts with import myapp , all of myapp/myapp.py's names will be in the myapp namespace. You should change that first line to be from myapp import * as you put further down in the question.
Mandatory note: You should avoid doing import * and variants in Python projects that import names from packages. Python programs and projects should always import individual names from packages/modules - with the name written explicitly, anyone reading your code will know exactly from were each function or class you use came from. Moreover, IDE's and other programming tools will be able to check for non-existing or inconsistent names and warn you.
However, note it is ok to use from module import * inside a package's __init__.py to get the names exposed (in the __all__ export) in each of the package's modules into the package namespace itself - which is what you want.
I am designing a python package with the following directory structure
package\
__init__.py
subpackage1\
__init__.py
module1.py
module2.py
subpackage2\
__init__.py
module3.py
I would like users to be able to explore a clean name space that reflects the directory structure using tab completion when importing the package in ipython.
For example, after doing
import package as pkg
I want tab completion on pkg. to show pkg.subpackage1 pkg.subpackage2 and tab completion on pkg.subpackage1. to show pkg.subpackage1.module1 pgk.subpackage1.module2. Some of these modules depend on each other, and include import statements.
For example in module1.py we have,
from ..subpackage2 import module3
However, I don't want someone to be able to tab complete the following pkg.subpackage1.module1.module3 even if they've done import pkg.subpackage1.module1.
In addition, when people do import pkg.subpackage1.module1 I don't want tab completion on pkg.subpackage1.module1. to show things like my internal exception classes and the fact that I imported numpy as np in module1. In other words, I'd like my usage of module3 in module1 to be hidden from the user as well as my usage of numpy as np. Is using things like import numpy as _np and from ..subpackage2 import module3 as _module3 the best way to do this?
Do I have to prepend an underscore to everything I don't want them to see?
To clarify, I can see in the scipy source code that the file scipy.integrate.quadrature has the line import numpy as np in it, but when I do import scipy in ipython I can tab complete out to scipy.integrate.quadrature and not see np
In your package/__init__.py, include:
import subpackage1
import subpackage2
This makes sure anytime package is imported, it also imports subpackageX, as package.subpackageX.
In your subpackageX/__init__.py you don't include anything. So in order to get the package.subpackageX.moduleX defined, one would have to import it explicitly (e.g. from package.subpackage1 import module1)
Note that if you take the underscores approach, in ipython, one could still tab-complete the names prefixed with underscores, if doing package.subpackage1._<TAB>.
EDIT:
Other alternatives:
In subpackage1/__init__.py, import the names you want to expose,
e.g. from .module1 import x,y. That would make them defined as
package.subpackage1.x, not package.subpackage1.module1.x
Define the names using the __all__ directive. That would not affect tab-completion, but would be a declarative way of saying what the importer should be of interest. This way, they can also do from package.subpackage1 import *.
I'm currently coding an app which is basically structured that way :
main.py
+ Package1
+--- Class1.py
+--- Apps
+ Package2
+--- Class1.py
+--- Apps
So I have two questions :
First, inside both packages, there are modules needed by all Apps, eg : re. Is there a way I can import a module for the whole package at once, instead of importing it in every file that needs it ?
And, as you can see, Class1 is used in both packages. Is there a good way to share it between both packages to avoid code duplication ?
I would strongly recommend against doing this: by separating the imports from the module that uses the functionality, you make it more difficult to track dependencies between modules.
If you really want to do it though, one option would be to create a new module called common_imports (for example) and have it do the imports you are after.
Then in your other modules, add the following:
from common_imports import *
This should give you all the public names from that module (including all the imports).
To answer your second question, if your two modules named Class1.py are in fact the same, then you should not copy it to both packages. Place it in a package which will contain only code which is common to both, and then import it. It is absolutely not necessary to copy the file and try to maintain each change in both copies.
Q1:
You Must find a way to import your package, thus you have two choices:
(Please correct me if I'm wrong or not thorough)
1. Look at James' solution, which you need to define a class, put all the modules inside and finally import them to your Sub-classes
2. Basically import nothing to your Main class, but instead, import only once to your Sub-classes
For example:(inside A_1 subclass)
import re
def functionThatUseRe(input):
pass
Then inside your main class, just do
try:
from YourPackage import A_1 #windows
except:
import A_1 #MAC OSX
A_1.functionThatUseRe("")
And you completely avoided importing modules multiple times
Q2: put your class1.py in the same directory with your main class, or move it to another folder, in Package1(&2).Apps
import Class1
Start using the code from there
I have a directory structure that looks like this:
project/
__init__.py
foo/
__init.py__
first.py
second.py
third.py
plum.py
In project/foo/__init__.py I import classes from first.py, second.py and third.py and put them in __all__.
There's a class in first.py named WonderfulThing which I'd like to use in second.py, and want to import by importing * from foo. (It's outside of the scope of this question why I'd like to do so, assume I have a good reason.)
In second.py I've tried from .foo import *, from foo import * and from . import * and in none of these cases is WonderfulThing imported. I also tried from ..foo import *, which raises an error "Attempted relative import beyond toplevel package".
I've read the docs and the PEP, and I can't work out how to make this work. Any assistance would be appreciated.
Clarification/Edit: It seems like I may have been misunderstanding the way __all__ works in packages. I was using it the same as in modules,
from .first import WonderfulThing
__all__ = [ "WonderfulThing" ]
but looking at the docs again it seems to suggest that __all__ may only be used in packages to specify the names of modules to be imported by default; there doesn't seem to be any way to include anything that's not a module.
Is this correct?
A non-wildcard import failed (cannot import name WonderfulThing). Trying from . import foo failed, but import foo works. Unfortunately, dir(foo) shows nothing.
Edit: I did misunderstand the question: No __all__ is not restricted to just modules.
One question is why you want to do a relative import. There is nothing wrong with doing from project.foo import *, here. Secondly, the __all__ restriction on foo won't prevent you from doing from project.foo.first import WonderfulThing, or just from .first import WonderfulThing, which still will be the best way.
And if you really want to import a a lot of things, it's probably best to do from project import foo, and then use the things with foo.WonderfulThing instead for doing an import * and then using WonderfulThing directly.
However to answer your direct question, to import from the __init__ file in second.py you do this:
from . import WonderfulThing
or
from . import *