I have a python package of the form:
package
├── __init__.py
├── module_1.py
└── module_2.py
Inside __init__.py I have:
__all__ = ["module_1", "module_2"]
Module_1 is:
__all__ = ["foo"]
def foo(): pass
Module_2 is:
__all__ = ["Bar"]
class Bar: pass
From the documentation I thought that the following code would import foo and Bar:
import package as pk
However, when I run pk.foo() it throws an error: AttributeError: module 'package' has no attribute 'foo' (same for Bar).
Thanks to this answer, I know that to get the desired behavior, I can change __init__.py into:
from .module_1 import *
from .module_2 import *
The above works.
However, I do not understand the documentation. The lines:
if a package’s __init__.py code defines a list named __all__, it is taken to be the list of module names that should be imported when from package import * is encountered
Sound like my original __init__.py should have worked (the one with __all__ = ["module_1", "module_2"]). That is, the line import package as pk should have imported the modules module_1 and module_2, which in turn make foo and Bar available.
What am I missing?
EDIT:
I have also tried using exactly what the documentation mentions. That is, using from package import *, and then tried with package.foo() and foo(), but neither worked.
The first, package.foo(), throws the error NameError: 'package' is not defined.. The second, foo(), throws the error NameError: 'foo' is not defined..
How would a working example look like?.. One where __init__.py is of the form __all__ = ["module_1", "module_2"].
I will try to summarize the knowledge gained from the comments to my question, the documentation, my own tests, and this post.
1) __all__ behaves differently on __init__ and modules
1.1) Within a module
When __all__ is within a module, it determines what objects are made available when running from module import *.
Given this package structure:
package
├── __init__.py
├── module_1.py
└── module_2.py
And given the following code for module_1:
__all__ = ["foo"]
def foo(): pass
def baz(): pass
Running from package.module_1 import * will make foo available but not baz.
Moreover, foo can then be called using foo(), i.e., there is no need to reference the module.
1.2) Within __init__ (my original question)
When __all__ is within __init__,
it is taken to be the list of module names that should be imported when from package import * is encountered.
Running from package import * will then have two effects:
The scripts of the modules in __all__ will be ran (they are imported).
These modules are made available in the namespace.
This means that if __init__ is of the form:
__all__ = ["module_1", "module_2"]
Then, running from package import * will run the scripts of module_1 and module_2 and make both modules available. So, now, the function foo inside module_1 can be called as module_1.foo() instead of package.module_1.foo().
However, if this is the intent, then using from package.module_1 import foo might be better. As it makes foo accessible as foo().
2) from package import * is not the same as import package
Running from package import * has the effect mentioned in 1.2). However, this is not true for running import package: i.e. module_1.foo() would not work in this scenario.
3) The alternative approach to __init__
(The following is based on this post) As I mentioned in my question, there is an alternative approach to __init__, in which the objects that you want to make available when the user calls from package import * are directly imported into __init__.
As an example, __init__ could contain the following code:
from .module_1 import *
from .module_2 import *
Then, when the user calls from package import * the objects from modules 1 and 2 would be available on the namespace.
If module_1 was that of 1.1), then the function foo could be called without referencing the module. i.e. foo(). However, the same would not work for baz.
As mentioned in 2), from package import * is different to import package. Calling import package in this scenario (with this __init__), makes foo available through package.foo(), instead of just foo(). Similarly import package as pk makes foo available as pk.foo().
This approach could be preferable to that of 1.2), in which foo is made available through module_1.foo().
The key problem in your code is that you have defined __all__ in sub modules. Which allows to be exported only that module, but such module is not available in your directory. They are definitions you want to use.
So, just remove them from sub modules and you'll get it working.
(Python 3.6)
I have this folder structure:
package/
start.py
subpackage/
__init__.py
submodule.py
submodule.py:
def subfunc():
print("This is submodule")
__ init __.py:
from subpackage.submodule import subfunc
start.py:
import subpackage
subpackage.subfunc()
subpackage.submodule.subfunc()
I understand how and why
subpackage.subfunc()
works.
But I don't understand why:
subpackage.submodule.subfunc()
also works, if I have not done:
from subpackage import submodule
Nor:
import subpackage.submodule
Neither in __ init __.py nor in start.py
Thank you very much if anyone may clear my doubt.
When issuing from subpackage.submodule import subfunc, python does two things for you: one, search and evaluate the module named subpackage.submodule, put it into sys.modules cache; two, populate subpackage.submodule.subfunc object and bind name "subfunc" to the namespace of the current module:
The import statement combines two operations; it searches for the named module, then it binds the results of that search to a name in the local scope.
When importing subpackage.submodule, parent of submodule also got imported:
While certain side-effects may occur, such as the importing of parent packages, and the updating of various caches (including sys.modules) ...
On the last stage of importing subpackage.submodule, python would set the module as an attribute on its parent subpackage, this behavior is documented:
When a submodule is loaded using any mechanism (e.g. importlib APIs, the import or import-from statements, or built-in __import__()) a binding is placed in the parent module’s namespace to the submodule object.
If I'm getting this right, you have a folder called "package" in which there are 2 things: a .py file and another folder called "subpackage".
Inside "subpackage" you have __init__.py and submodule.py which the latter contains a function that just prints "This is submodule".
Now, when you call import subpackage, you call and "pull" everything that's inside "subpackage", including submodule and therefore, the subfunc() function.
When you write subpackage.submodule.subfunc() there's really nothing amazing going there, you just call the mainfolder/container (subpackage.), then the .py file (submodule.) and finally the function itself (subfunc() ).
The Python documentation says
Consider this code:
import sound.effects.echo
import sound.effects.surround
from sound.effects import *
In this example, the echo and surround modules are imported in the
current namespace because they are defined in the sound.effects
package when the from...import statement is executed. (This also works
when __all__ is defined.)
I try the following code
# package/
# __init__.py
# sub_module.py
import package.sub_module
from package import *
print(sub_module)
When package/__init__.py is empty, the code works fine. However, when package/__init__.py contains __all__ = [], print(sub_module) will raise NameError.
What is (This also works when all is defined.) from the documentation means?
The codes:
package/
__init__.py
sub_module.py # empty file
main.py
In main.py:
import package.sub_module
from package import *
print(sub_module)
When package/__init__.py is empty, executing python3 main.py gets
<module 'package.sub_module' from '/path/to/package/sub_module.py'
When package/__init__.py contains __all__ = [], executing python3 main.py gets
Traceback (most recent call last):
File "main.py", line 3, in <module>
print(sub_module)
NameError: name 'sub_module' is not defined
If a module package defines __all__, it is the list of module names that are imported by from package import *
So if you define __all__ as empty list, from package import * will import nothing.
Try defining it like this:
__all__ = ['sub_module']
Also note that you don't have to do from package import * to use sub_module
You can also just do:
import package.sub_module
print(package.sub_module)
Solution: you have __all__ set to empty list i.e. from package import * basically imports nothing
set it to __all__ = ['submodule'] in __init__.py
What exactly is __all__ ?
In simplest words all help customizing the from package import * i.e. with all we can set what will be imported and what not.
From the docs:
The public names defined by a module are determined by checking the
module’s namespace for a variable named all; if defined, it must
be a sequence of strings which are names defined or imported by that
module. The names given in all are all considered public and are
required to exist. If all is not defined, the set of public names
includes all names found in the module’s namespace which do not begin
with an underscore character ('_'). all should contain the entire
public API. It is intended to avoid accidentally exporting items that
are not part of the API (such as library modules which were imported
and used within the module).
One important thing to note here is - Imports without * are not affected by __all__ i.e. Members that are not mentioned in __all__ are accessible from outside the module using direct import - from <module> import <member>.
An Example: the following code in a module.py explicitly exports the symbols foo and bar:
__all__ = ['foo', 'bar']
waz = 5
foo = 10
def bar(): return 'bar'
These symbols can then be imported like so:
from foo import *
print foo
print bar
# now import `waz` will trigger an exception,
# as it is not in the `__all__`, hence not a public member.
print waz
If you define __all__, then only the attributes mentioned there will be imported via *, while the excluded ones have to be imported explicitly. So either use
from package import submodule
or if you really want to use the (discouraged!) from package import *, declare
__all__ = ['submodule']
in package. Note how tedious it will become to keep this up-to-date...
This is my project structure (Python 3.5.1.):
a
├── b.py
└── __init__.py
Case 1
File b.py is empty.
File __init__.py is:
print(b)
If we run import a, the output is:
NameError: name 'b' is not defined
Case 2
File b.py is empty.
File __init__.py is:
import a.b
print(b)
If we run import a, the output is:
<module 'a.b' from '/tmp/a/b.py'>
Question
Why doesn't the program fail in Case 2?
Usually if we run import a.b then we can only reference it by a.b, not b. Hopefully somebody can help explain what's happening to the namespace in Case 2.
Python adds modules as globals to the parent package after import.
So when you imported a.b, the name b was added as a global to the a module, created by a/__init__.py.
From the Python 3 import system documentation:
When a submodule is loaded using any mechanism (e.g. importlib APIs, the import or import-from statements, or built-in __import__()) a binding is placed in the parent module’s namespace to the submodule object. For example, if package spam has a submodule foo, after importing spam.foo, spam will have an attribute foo which is bound to the submodule.
Bold emphasis mine. Note that the same applies to Python 2, but Python 3 made the process more explicit.
An import statement brings a module into scope. You imported b, so there it is, a module object.
Read the documentation for import:
The basic import statement (no from clause) is executed in two steps:
find a module, loading and initializing it if necessary
define a name or names in the local namespace for the scope where the import statement occurs.
You didn't import b in the first case.
I see __all__ in __init__.py files. What does it do?
Linked to, but not explicitly mentioned here, is exactly when __all__ is used. It is a list of strings defining what symbols in a module will be exported when from <module> import * is used on the module.
For example, the following code in a foo.py explicitly exports the symbols bar and baz:
__all__ = ['bar', 'baz']
waz = 5
bar = 10
def baz(): return 'baz'
These symbols can then be imported like so:
from foo import *
print(bar)
print(baz)
# The following will trigger an exception, as "waz" is not exported by the module
print(waz)
If the __all__ above is commented out, this code will then execute to completion, as the default behaviour of import * is to import all symbols that do not begin with an underscore, from the given namespace.
Reference: https://docs.python.org/tutorial/modules.html#importing-from-a-package
NOTE: __all__ affects the from <module> import * behavior only. Members that are not mentioned in __all__ are still accessible from outside the module and can be imported with from <module> import <member>.
It's a list of public objects of that module, as interpreted by import *. It overrides the default of hiding everything that begins with an underscore.
Explain all in Python?
I keep seeing the variable __all__ set in different __init__.py files.
What does this do?
What does __all__ do?
It declares the semantically "public" names from a module. If there is a name in __all__, users are expected to use it, and they can have the expectation that it will not change.
It also will have programmatic effects:
import *
__all__ in a module, e.g. module.py:
__all__ = ['foo', 'Bar']
means that when you import * from the module, only those names in the __all__ are imported:
from module import * # imports foo and Bar
Documentation tools
Documentation and code autocompletion tools may (in fact, should) also inspect the __all__ to determine what names to show as available from a module.
__init__.py makes a directory a Python package
From the docs:
The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path.
In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable.
So the __init__.py can declare the __all__ for a package.
Managing an API:
A package is typically made up of modules that may import one another, but that are necessarily tied together with an __init__.py file. That file is what makes the directory an actual Python package. For example, say you have the following files in a package:
package
├── __init__.py
├── module_1.py
└── module_2.py
Let's create these files with Python so you can follow along - you could paste the following into a Python 3 shell:
from pathlib import Path
package = Path('package')
package.mkdir()
(package / '__init__.py').write_text("""
from .module_1 import *
from .module_2 import *
""")
package_module_1 = package / 'module_1.py'
package_module_1.write_text("""
__all__ = ['foo']
imp_detail1 = imp_detail2 = imp_detail3 = None
def foo(): pass
""")
package_module_2 = package / 'module_2.py'
package_module_2.write_text("""
__all__ = ['Bar']
imp_detail1 = imp_detail2 = imp_detail3 = None
class Bar: pass
""")
And now you have presented a complete api that someone else can use when they import your package, like so:
import package
package.foo()
package.Bar()
And the package won't have all the other implementation details you used when creating your modules cluttering up the package namespace.
__all__ in __init__.py
After more work, maybe you've decided that the modules are too big (like many thousands of lines?) and need to be split up. So you do the following:
package
├── __init__.py
├── module_1
│ ├── foo_implementation.py
│ └── __init__.py
└── module_2
├── Bar_implementation.py
└── __init__.py
First make the subpackage directories with the same names as the modules:
subpackage_1 = package / 'module_1'
subpackage_1.mkdir()
subpackage_2 = package / 'module_2'
subpackage_2.mkdir()
Move the implementations:
package_module_1.rename(subpackage_1 / 'foo_implementation.py')
package_module_2.rename(subpackage_2 / 'Bar_implementation.py')
create __init__.pys for the subpackages that declare the __all__ for each:
(subpackage_1 / '__init__.py').write_text("""
from .foo_implementation import *
__all__ = ['foo']
""")
(subpackage_2 / '__init__.py').write_text("""
from .Bar_implementation import *
__all__ = ['Bar']
""")
And now you still have the api provisioned at the package level:
>>> import package
>>> package.foo()
>>> package.Bar()
<package.module_2.Bar_implementation.Bar object at 0x7f0c2349d210>
And you can easily add things to your API that you can manage at the subpackage level instead of the subpackage's module level. If you want to add a new name to the API, you simply update the __init__.py, e.g. in module_2:
from .Bar_implementation import *
from .Baz_implementation import *
__all__ = ['Bar', 'Baz']
And if you're not ready to publish Baz in the top level API, in your top level __init__.py you could have:
from .module_1 import * # also constrained by __all__'s
from .module_2 import * # in the __init__.py's
__all__ = ['foo', 'Bar'] # further constraining the names advertised
and if your users are aware of the availability of Baz, they can use it:
import package
package.Baz()
but if they don't know about it, other tools (like pydoc) won't inform them.
You can later change that when Baz is ready for prime time:
from .module_1 import *
from .module_2 import *
__all__ = ['foo', 'Bar', 'Baz']
Prefixing _ versus __all__:
By default, Python will export all names that do not start with an _ when imported with import *. As demonstrated by the shell session here, import * does not bring in the _us_non_public name from the us.py module:
$ cat us.py
USALLCAPS = "all caps"
us_snake_case = "snake_case"
_us_non_public = "shouldn't import"
$ python
Python 3.10.0 (default, Oct 4 2021, 17:55:55) [GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from us import *
>>> dir()
['USALLCAPS', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'us_snake_case']
You certainly could rely on this mechanism. Some packages in the Python standard library, in fact, do rely on this, but to do so, they alias their imports, for example, in ctypes/__init__.py:
import os as _os, sys as _sys
Using the _ convention can be more elegant because it removes the redundancy of naming the names again. But it adds the redundancy for imports (if you have a lot of them) and it is easy to forget to do this consistently - and the last thing you want is to have to indefinitely support something you intended to only be an implementation detail, just because you forgot to prefix an _ when naming a function.
I personally write an __all__ early in my development lifecycle for modules so that others who might use my code know what they should use and not use.
Most packages in the standard library also use __all__.
When avoiding __all__ makes sense
It makes sense to stick to the _ prefix convention in lieu of __all__ when:
You're still in early development mode and have no users, and are constantly tweaking your API.
Maybe you do have users, but you have unittests that cover the API, and you're still actively adding to the API and tweaking in development.
An export decorator
The downside of using __all__ is that you have to write the names of functions and classes being exported twice - and the information is kept separate from the definitions. We could use a decorator to solve this problem.
I got the idea for such an export decorator from David Beazley's talk on packaging. This implementation seems to work well in CPython's traditional importer. If you have a special import hook or system, I do not guarantee it, but if you adopt it, it is fairly trivial to back out - you'll just need to manually add the names back into the __all__
So in, for example, a utility library, you would define the decorator:
import sys
def export(fn):
mod = sys.modules[fn.__module__]
if hasattr(mod, '__all__'):
mod.__all__.append(fn.__name__)
else:
mod.__all__ = [fn.__name__]
return fn
and then, where you would define an __all__, you do this:
$ cat > main.py
from lib import export
__all__ = [] # optional - we create a list if __all__ is not there.
#export
def foo(): pass
#export
def bar():
'bar'
def main():
print('main')
if __name__ == '__main__':
main()
And this works fine whether run as main or imported by another function.
$ cat > run.py
import main
main.main()
$ python run.py
main
And API provisioning with import * will work too:
$ cat > run.py
from main import *
foo()
bar()
main() # expected to error here, not exported
$ python run.py
Traceback (most recent call last):
File "run.py", line 4, in <module>
main() # expected to error here, not exported
NameError: name 'main' is not defined
I'm just adding this to be precise:
All other answers refer to modules. The original question explicitely mentioned __all__ in __init__.py files, so this is about python packages.
Generally, __all__ only comes into play when the from xxx import * variant of the import statement is used. This applies to packages as well as to modules.
The behaviour for modules is explained in the other answers. The exact behaviour for packages is described here in detail.
In short, __all__ on package level does approximately the same thing as for modules, except it deals with modules within the package (in contrast to specifying names within the module). So __all__ specifies all modules that shall be loaded and imported into the current namespace when us use from package import *.
The big difference is, that when you omit the declaration of __all__ in a package's __init__.py, the statement from package import * will not import anything at all (with exceptions explained in the documentation, see link above).
On the other hand, if you omit __all__ in a module, the "starred import" will import all names (not starting with an underscore) defined in the module.
It also changes what pydoc will show:
module1.py
a = "A"
b = "B"
c = "C"
module2.py
__all__ = ['a', 'b']
a = "A"
b = "B"
c = "C"
$ pydoc module1
Help on module module1:
NAME
module1
FILE
module1.py
DATA
a = 'A'
b = 'B'
c = 'C'
$ pydoc module2
Help on module module2:
NAME
module2
FILE
module2.py
DATA
__all__ = ['a', 'b']
a = 'A'
b = 'B'
I declare __all__ in all my modules, as well as underscore internal details, these really help when using things you've never used before in live interpreter sessions.
__all__ customizes * in from <module> import *
and from <package> import *.
A module is a .py file meant to be imported.
A package is a directory with a __init__.py file. A package usually contains modules.
MODULES
""" cheese.py - an example module """
__all__ = ['swiss', 'cheddar']
swiss = 4.99
cheddar = 3.99
gouda = 10.99
__all__ lets humans know the "public" features of a module.[#AaronHall] Also, pydoc recognizes them.[#Longpoke]
from module import *
See how swiss and cheddar are brought into the local namespace, but not gouda:
>>> from cheese import *
>>> swiss, cheddar
(4.99, 3.99)
>>> gouda
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'gouda' is not defined
Without __all__, any symbol (that doesn't start with an underscore) would have been available.
Imports without * are not affected by __all__
import module
>>> import cheese
>>> cheese.swiss, cheese.cheddar, cheese.gouda
(4.99, 3.99, 10.99)
from module import names
>>> from cheese import swiss, cheddar, gouda
>>> swiss, cheddar, gouda
(4.99, 3.99, 10.99)
import module as localname
>>> import cheese as ch
>>> ch.swiss, ch.cheddar, ch.gouda
(4.99, 3.99, 10.99)
PACKAGES
In the __init__.py file of a package __all__ is a list of strings with the names of public modules or other objects. Those features are available to wildcard imports. As with modules, __all__ customizes the * when wildcard-importing from the package.[#MartinStettner]
Here's an excerpt from the Python MySQL Connector __init__.py:
__all__ = [
'MySQLConnection', 'Connect', 'custom_error_exception',
# Some useful constants
'FieldType', 'FieldFlag', 'ClientFlag', 'CharacterSet', 'RefreshOption',
'HAVE_CEXT',
# Error handling
'Error', 'Warning',
...etc...
]
The default case, asterisk with no __all__ for a package, is complicated, because the obvious behavior would be expensive: to use the file system to search for all modules in the package. Instead, in my reading of the docs, only the objects defined in __init__.py are imported:
If __all__ is not defined, the statement from sound.effects import * does not import all submodules from the package sound.effects into the current namespace; it only ensures that the package sound.effects has been imported (possibly running any initialization code in __init__.py) and then imports whatever names are defined in the package. This includes any names defined (and submodules explicitly loaded) by __init__.py. It also includes any submodules of the package that were explicitly loaded by previous import statements.
And lastly, a venerated tradition for stack overflow answers, professors, and mansplainers everywhere, is the bon mot of reproach for asking a question in the first place:
Wildcard imports ... should be avoided, as they [confuse] readers and many automated tools.
[PEP 8, #ToolmakerSteve]
Short answer
__all__ affects from <module> import * statements.
Long answer
Consider this example:
foo
├── bar.py
└── __init__.py
In foo/__init__.py:
(Implicit) If we don't define __all__, then from foo import * will only import names defined in foo/__init__.py.
(Explicit) If we define __all__ = [], then from foo import * will import nothing.
(Explicit) If we define __all__ = [ <name1>, ... ], then from foo import * will only import those names.
Note that in the implicit case, python won't import names starting with _. However, you can force importing such names using __all__.
You can view the Python document here.
__all__ is used to document the public API of a Python module. Although it is optional, __all__ should be used.
Here is the relevant excerpt from the Python language reference:
The public names defined by a module are determined by checking the module’s namespace for a variable named __all__; if defined, it must be a sequence of strings which are names defined or imported by that module. The names given in __all__ are all considered public and are required to exist. If __all__ is not defined, the set of public names includes all names found in the module’s namespace which do not begin with an underscore character ('_'). __all__ should contain the entire public API. It is intended to avoid accidentally exporting items that are not part of the API (such as library modules which were imported and used within the module).
PEP 8 uses similar wording, although it also makes it clear that imported names are not part of the public API when __all__ is absent:
To better support introspection, modules should explicitly declare the names in their public API using the __all__ attribute. Setting __all__ to an empty list indicates that the module has no public API.
[...]
Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as os.path or a package's __init__ module that exposes functionality from submodules.
Furthermore, as pointed out in other answers, __all__ is used to enable wildcard importing for packages:
The import statement uses the following convention: if a package’s __init__.py code defines a list named __all__, it is taken to be the list of module names that should be imported when from package import * is encountered.
__all__ affects how from foo import * works.
Code that is inside a module body (but not in the body of a function or class) may use an asterisk (*) in a from statement:
from foo import *
The * requests that all attributes of module foo (except those beginning with underscores) be bound as global variables in the importing module. When foo has an attribute __all__, the attribute's value is the list of the names that are bound by this type of from statement.
If foo is a package and its __init__.py defines a list named __all__, it is taken to be the list of submodule names that should be imported when from foo import * is encountered. If __all__ is not defined, the statement from foo import * imports whatever names are defined in the package. This includes any names defined (and submodules explicitly loaded) by __init__.py.
Note that __all__ doesn't have to be a list. As per the documentation on the import statement, if defined, __all__ must be a sequence of strings which are names defined or imported by the module. So you may as well use a tuple to save some memory and CPU cycles. Just don't forget a comma in case the module defines a single public name:
__all__ = ('some_name',)
See also Why is “import *” bad?
This is defined in PEP8 here:
Global Variable Names
(Let's hope that these variables are meant for use inside one module only.) The conventions are about the same as those for functions.
Modules that are designed for use via from M import * should use the __all__ mechanism to prevent exporting globals, or use the older convention of prefixing such globals with an underscore (which you might want to do to indicate these globals are "module non-public").
PEP8 provides coding conventions for the Python code comprising the standard library in the main Python distribution. The more you follow this, closer you are to the original intent.