I am designing a python package with the following directory structure
package\
__init__.py
subpackage1\
__init__.py
module1.py
module2.py
subpackage2\
__init__.py
module3.py
I would like users to be able to explore a clean name space that reflects the directory structure using tab completion when importing the package in ipython.
For example, after doing
import package as pkg
I want tab completion on pkg. to show pkg.subpackage1 pkg.subpackage2 and tab completion on pkg.subpackage1. to show pkg.subpackage1.module1 pgk.subpackage1.module2. Some of these modules depend on each other, and include import statements.
For example in module1.py we have,
from ..subpackage2 import module3
However, I don't want someone to be able to tab complete the following pkg.subpackage1.module1.module3 even if they've done import pkg.subpackage1.module1.
In addition, when people do import pkg.subpackage1.module1 I don't want tab completion on pkg.subpackage1.module1. to show things like my internal exception classes and the fact that I imported numpy as np in module1. In other words, I'd like my usage of module3 in module1 to be hidden from the user as well as my usage of numpy as np. Is using things like import numpy as _np and from ..subpackage2 import module3 as _module3 the best way to do this?
Do I have to prepend an underscore to everything I don't want them to see?
To clarify, I can see in the scipy source code that the file scipy.integrate.quadrature has the line import numpy as np in it, but when I do import scipy in ipython I can tab complete out to scipy.integrate.quadrature and not see np
In your package/__init__.py, include:
import subpackage1
import subpackage2
This makes sure anytime package is imported, it also imports subpackageX, as package.subpackageX.
In your subpackageX/__init__.py you don't include anything. So in order to get the package.subpackageX.moduleX defined, one would have to import it explicitly (e.g. from package.subpackage1 import module1)
Note that if you take the underscores approach, in ipython, one could still tab-complete the names prefixed with underscores, if doing package.subpackage1._<TAB>.
EDIT:
Other alternatives:
In subpackage1/__init__.py, import the names you want to expose,
e.g. from .module1 import x,y. That would make them defined as
package.subpackage1.x, not package.subpackage1.module1.x
Define the names using the __all__ directive. That would not affect tab-completion, but would be a declarative way of saying what the importer should be of interest. This way, they can also do from package.subpackage1 import *.
Related
I currently have a module I created that has a number of functions.
It's getting quite large so I figured I should make it into a package and split the functions up to make it more manageable.
I'm just testing out how this all works before I do this for real so apologies if it seems a bit tenuous.
I've created a folder called pack_test and in it I have:
__init__.py
foo.py
bar.py
__init__.py contains:
__all__ = ['foo', 'bar']
from . import *
import subprocess
from os import environ
In the console I can write import pack_test as pt and this is fine, no errors.
pt. and two tabs shows me that I can see pt.bar, pt.environ, pt.foo and pt.subprocess in there.
All good so far.
If I want to reference subprocess or environ in foo.py or bar.py how do I do it in there?
If in bar.py I have a function which just does return subprocess.call('ls') it errors saying NameError: name 'subprocess' is not defined. There must be something I'm missing which enables me to reference subprocess from the level above? Presumably, once I can get the syntax from that I can also just call environ in a similar way?
The alternative as I could see it would be to have import subprocess in both foo.py and bar.py but then this seems a bit odd to me to have it appear across multiple files when I could have it the once at a higher level, particularly if I went on to have a large number of files rather than just 2 in this example.
TL;DR:
__init__.py :
import foo
import bar
__all__ = ["foo", "bar"]
foo.py:
import subprocess
from os import environ
# your code here
bar.py
import subprocess
from os import environ
# your code here
There must be something I'm missing which enables me to reference subprocess from the level above?
Nope, this is the expected behaviour.
import loads a module (if it isn't already), caches it in sys.modules (idem), and bind the imported names in the current namespace. Each Python module has (or "is") it's own namespace (there's no real "global" namespace). IOW, you have to import what you need in each module, ie if foo.py needs subprocess, it must explicitely import it.
This can seem a bit tedious at first but in the long run it really helps wrt/ maintainability - you just have to read the imports at the top of your module (pep 08: always put all imports at the beginning of the module) to know where a name comes from.
Also you should not use star imports (aka wild card imports aka from xxx import *) anywhere else than in your python shell (and even then...) - it's a maintainance time bomb. Not only because you don't know where each name comes from, but also because it's a sure way to rebind an already import name. Imagine that your foo module defines function "func". Somewhere you have "from foo import *; from bar import *", then later in the code a call to func. Now someone edits bar.py and adds a (distinct) "func" function, and suddenly you call fails, because you're not calling the expected "func". Now enjoy debugging this... And real-life examples are usually a bit more complex than this.
So if you fancy your mental sanity, don't be lazy, don't try to be smart either, just do the simple obvious thing: explicitely import the names you're interested in at the top of your modules.
(been here, done that etc)
You could create modules.py containing
import subprocess
import os
Then in foo.py or any of your files just have.
from modules import *
Your import statements in your files are then static and just update modules.py when you want to add an additional module accessible to them all.
I have the following package structure
package
__init__.py
sub1
__init__.py
foo.py # Contains class Foo
sub2
__init__.py
bar.py # Contains class Bar
I want to be able to just import package and have package.Foo and package.Bar, i.e. I want to have the subpackages be transparent to users.
The catch is that importing sub2 takes a long time, and many users don't care at all about the stuff in sub2 and only want the stuff in sub1. Thus I want users to be able to say import package.sub1 or from package import sub1 to just import sub1 and skip the import of sub2.
I know I can achieve the first part by having package/__init__.py contain
from .sub1 import *
from .sub2 import *
and having package/sub1/__init__.py be from .foo import Foo and similarly for sub2. However, this will always import sub1 and sub2 even if the user tries to import only package.sub1.
Correspondingly, I can achieve the second part by having package/__init__.py be empty and using the same sub1/__init__.py as above. However, then just saying import package doesn't load sub1 or sub2, so users would have to explicitly load them and then refer to package.sub1.Foo.
Ideally a solution would work both in 2.7.10 and 3.5.0, but I'll accept one or the other if both isn't possible.
The LazyLoader class is provided for exactly this kind of situation: postponing loading of the module when it is actually used, instead of at the point of importing it.
To build a lazy loader you can follow the example in the documentation:
suffixes = importlib.machinery.SOURCE_SUFFIXES
loader = importlib.machinery.SourceFileLoader
lazy_loader = importlib.util.LazyLoader.factory(loader)
finder = importlib.machinery.FileFinder(path, [(lazy_loader, suffixes)])
then you can use finder.find_spec to obtain the spec of a module and pass the result to Loader.create_module to load it.
This is a bit cumbersome to do manually for just one module.
Note that searching for "lazy import python" you'll find quite a few solutions that have different pro and cons, some of which run in python2.x. However the LazyLoader class above is the official way of doing it in python3.5+
you could add your shortcuts to the __init__.py of your module:
package/__init__.py
__all__ = [
... add everything you want to be listed for this module
'Foo',
'Bar',
...
]
from package.sub1.foo import Foo
from package.sub2.bar import Bar
now you should be able to call:
from package import Bar
Is it ok to use a module referencing with more than two dots in a path? Like in this example:
# Project structure:
# sound
# __init__.py
# codecs
# __init__.py
# echo
# __init__.py
# nix
# __init__.py
# way1.py
# way2.py
# way2.py source code
from .way1 import echo_way1
from ...codecs import cool_codec
# Do something with echo_way1 and cool_codec.
UPD: Changed the example. And I know, this will work in a practice. But is it a common method of importing or not?
update Nov. 24,2020
If you wanna dig deeper in python's relative-import, I strongly recommend you this answer.
Is it ok to use a module referencing with more than two dots in a path?
Yes. You can use multiple dots in relative import path, but it is only feasible when using from xxx import yyy syntax, not import xxx syntax. Moreover, single dot, two dots and three dots mean current directory, parent directory and grandparent directory respectively, and so on.
And I know, this will work in a practice. But is it a common method of importing or not?
It depends. If your project has complex directory structure, using absolute import would be "disgusting". For example,
from sub1.sub2.sub3.sub4.sub5 import yourmethod
. In this case, using relative import will make your code clean and neat. Maybe look like
from ...sub5 import yourmethod
From PEP8:
Absolute imports are recommended, as they are usually more readable and tend to be better behaved (or at least give better error messages) if the import system is incorrectly configured (such as when a directory inside a package ends up on sys.path):
import mypkg.sibling
from mypkg import sibling
from mypkg.sibling import example
However, explicit relative imports are an acceptable alternative to absolute imports, especially when dealing with complex package layouts where using absolute imports would be unnecessarily verbose:
from . import sibling
from .sibling import example
Standard library code should avoid complex package layouts and always use absolute imports.
I'm currently coding an app which is basically structured that way :
main.py
+ Package1
+--- Class1.py
+--- Apps
+ Package2
+--- Class1.py
+--- Apps
So I have two questions :
First, inside both packages, there are modules needed by all Apps, eg : re. Is there a way I can import a module for the whole package at once, instead of importing it in every file that needs it ?
And, as you can see, Class1 is used in both packages. Is there a good way to share it between both packages to avoid code duplication ?
I would strongly recommend against doing this: by separating the imports from the module that uses the functionality, you make it more difficult to track dependencies between modules.
If you really want to do it though, one option would be to create a new module called common_imports (for example) and have it do the imports you are after.
Then in your other modules, add the following:
from common_imports import *
This should give you all the public names from that module (including all the imports).
To answer your second question, if your two modules named Class1.py are in fact the same, then you should not copy it to both packages. Place it in a package which will contain only code which is common to both, and then import it. It is absolutely not necessary to copy the file and try to maintain each change in both copies.
Q1:
You Must find a way to import your package, thus you have two choices:
(Please correct me if I'm wrong or not thorough)
1. Look at James' solution, which you need to define a class, put all the modules inside and finally import them to your Sub-classes
2. Basically import nothing to your Main class, but instead, import only once to your Sub-classes
For example:(inside A_1 subclass)
import re
def functionThatUseRe(input):
pass
Then inside your main class, just do
try:
from YourPackage import A_1 #windows
except:
import A_1 #MAC OSX
A_1.functionThatUseRe("")
And you completely avoided importing modules multiple times
Q2: put your class1.py in the same directory with your main class, or move it to another folder, in Package1(&2).Apps
import Class1
Start using the code from there
I have a directory structure that looks like this:
project/
__init__.py
foo/
__init.py__
first.py
second.py
third.py
plum.py
In project/foo/__init__.py I import classes from first.py, second.py and third.py and put them in __all__.
There's a class in first.py named WonderfulThing which I'd like to use in second.py, and want to import by importing * from foo. (It's outside of the scope of this question why I'd like to do so, assume I have a good reason.)
In second.py I've tried from .foo import *, from foo import * and from . import * and in none of these cases is WonderfulThing imported. I also tried from ..foo import *, which raises an error "Attempted relative import beyond toplevel package".
I've read the docs and the PEP, and I can't work out how to make this work. Any assistance would be appreciated.
Clarification/Edit: It seems like I may have been misunderstanding the way __all__ works in packages. I was using it the same as in modules,
from .first import WonderfulThing
__all__ = [ "WonderfulThing" ]
but looking at the docs again it seems to suggest that __all__ may only be used in packages to specify the names of modules to be imported by default; there doesn't seem to be any way to include anything that's not a module.
Is this correct?
A non-wildcard import failed (cannot import name WonderfulThing). Trying from . import foo failed, but import foo works. Unfortunately, dir(foo) shows nothing.
Edit: I did misunderstand the question: No __all__ is not restricted to just modules.
One question is why you want to do a relative import. There is nothing wrong with doing from project.foo import *, here. Secondly, the __all__ restriction on foo won't prevent you from doing from project.foo.first import WonderfulThing, or just from .first import WonderfulThing, which still will be the best way.
And if you really want to import a a lot of things, it's probably best to do from project import foo, and then use the things with foo.WonderfulThing instead for doing an import * and then using WonderfulThing directly.
However to answer your direct question, to import from the __init__ file in second.py you do this:
from . import WonderfulThing
or
from . import *