I am trying to use automodapi to generate documentation for my Django project. When I call automodapi like this:
.. automodapi:: mypackage.mymodule
the output includes all imported classes and functions, e.g, the Django Model class, in the index of Functions and Classes. I would like to exclude the imports and only list those classes and functions declared in the module I specified.
I couldn't see anything about this in the documentation.
Is there a way to do this, preferably without modifying modules?
UPDATE: #saimn has provided a working solution using __all__ but my project doesn't use __all__. It would be nice if there was a solution that didn't involve modifying the modules.
Patching automodapi to only include locals also addresses this issue while not requiring any changes to your code:
def patch_automodapi(app):
"""Monkey-patch the automodapi extension to exclude imported members"""
from sphinx_automodapi import automodsumm
from sphinx_automodapi.utils import find_mod_objs
automodsumm.find_mod_objs = lambda *args: find_mod_objs(args[0], onlylocals=True)
def setup(app):
app.connect("builder-inited", patch_automodapi)
Source: https://github.com/astropy/sphinx-automodapi/issues/119
The above snippet goes into your conf.py sphinx configuration file.
You can use the __all__ variable (this should probably be stated more clearly in the documentation).
Related
A proper Python module will list all its public symbols in a list called __all__. Managing that list can be tedious, since you'll have to list each symbol twice. Surely there are better ways, probably using decorators so one would merely annotate the exported symbols as #export.
How would you write such a decorator? I'm certain there are different ways, so I'd like to see several answers with enough information that users can compare the approaches against one another.
In Is it a good practice to add names to __all__ using a decorator?, Ed L suggests the following, to be included in some utility library:
import sys
def export(fn):
"""Use a decorator to avoid retyping function/class names.
* Based on an idea by Duncan Booth:
http://groups.google.com/group/comp.lang.python/msg/11cbb03e09611b8a
* Improved via a suggestion by Dave Angel:
http://groups.google.com/group/comp.lang.python/msg/3d400fb22d8a42e1
"""
mod = sys.modules[fn.__module__]
if hasattr(mod, '__all__'):
name = fn.__name__
all_ = mod.__all__
if name not in all_:
all_.append(name)
else:
mod.__all__ = [fn.__name__]
return fn
We've adapted the name to match the other examples. With this in a local utility library, you'd simply write
from .utility import export
and then start using #export. Just one line of idiomatic Python, you can't get much simpler than this. On the downside, the module does require access to the module by using the __module__ property and the sys.modules cache, both of which may be problematic in some of the more esoteric setups (like custom import machinery, or wrapping functions from another module to create functions in this module).
The python part of the atpublic package by Barry Warsaw does something similar to this. It offers some keyword-based syntax, too, but the decorator variant relies on the same patterns used above.
This great answer by Aaron Hall suggests something very similar, with two more lines of code as it doesn't use __dict__.setdefault. It might be preferable if manipulating the module __dict__ is problematic for some reason.
You could simply declare the decorator at the module level like this:
__all__ = []
def export(obj):
__all__.append(obj.__name__)
return obj
This is perfect if you only use this in a single module. At 4 lines of code (plus probably some empty lines for typical formatting practices) it's not overly expensive to repeat this in different modules, but it does feel like code duplication in those cases.
You could define the following in some utility library:
def exporter():
all = []
def decorator(obj):
all.append(obj.__name__)
return obj
return decorator, all
export, __all__ = exporter()
export(exporter)
# possibly some other utilities, decorated with #export as well
Then inside your public library you'd do something like this:
from . import utility
export, __all__ = utility.exporter()
# start using #export
Using the library takes two lines of code here. It combines the definition of __all__ and the decorator. So people searching for one of them will find the other, thus helping readers to quickly understand your code. The above will also work in exotic environments, where the module may not be available from the sys.modules cache or where the __module__ property has been tampered with or some such.
https://github.com/russianidiot/public.py has yet another implementation of such a decorator. Its core file is currently 160 lines long! The crucial points appear to be the fact that it uses the inspect module to obtain the appropriate module based on the current call stack.
This is not a decorator approach, but provides the level of efficiency I think you're after.
https://pypi.org/project/auto-all/
You can use the two functions provided with the package to "start" and "end" capturing the module objects that you want included in the __all__ variable.
from auto_all import start_all, end_all
# Imports outside the start and end functions won't be externally availab;e.
from pathlib import Path
def a_private_function():
print("This is a private function.")
# Start defining externally accessible objects
start_all(globals())
def a_public_function():
print("This is a public function.")
# Stop defining externally accessible objects
end_all(globals())
The functions in the package are trivial (a few lines), so could be copied into your code if you want to avoid external dependencies.
While other variants are technically correct to a certain extent, one might also be sure that:
if the target module already has __all__ declared, it is handled correctly;
target appears in __all__ only once:
# utils.py
import sys
from typing import Any
def export(target: Any) -> Any:
"""
Mark a module-level object as exported.
Simplifies tracking of objects available via wildcard imports.
"""
mod = sys.modules[target.__module__]
__all__ = getattr(mod, '__all__', None)
if __all__ is None:
__all__ = []
setattr(mod, '__all__', __all__)
elif not isinstance(__all__, list):
__all__ = list(__all__)
setattr(mod, '__all__', __all__)
target_name = target.__name__
if target_name not in __all__:
__all__.append(target_name)
return target
From the perspective of an external user of the module, are both necessary?
From my understanding, by correctly prefix hidden functions with an underscore it essentially does the same thing as explicitly define __all__, but I keep seeing developers doing both in their code. Why is that?
When importing from a module with from modulename import * names starting with underscores are indeed skipped.
However, a module rarely contains only public API objects. Usually you've made imports to support the code as well, and those names are global in the module as well. Without __all__, those names would be part of the import too.
In other words, unless you want to 'export' os in the following example you should use __all__:
import os
from .implementation import some_other_api_call
_module_path = os.path.dirname(os.path.abspath(__file__))
_template = open(os.path.join(_module_path, 'templates/foo_template.txt')).read()
VERSION = '1.0.0'
def make_bar(baz, ham, spam):
return _template.format(baz, ham, spam)
__all__ = ['some_other_api_call', 'make_bar']
because without the __all__ list, Python cannot distinguish between some_other_api_call and os here and divine which one should not be imported when using from ... import *.
You could work around this by renaming all your imports, so import os as _os, but that'd just make your code less readable.
And an explicit export list is always nice. Explicit is better than implicit, as the Zen of Python tells you.
I also use __all__: that explictly tells module users what you intend to export. Searching the module for names is tedious, even if you are careful to do, e.g., import os as _os, etc. A wise man once wrote "explicit is better than implicit" ;-)
Defining all will overide the default behaviour. There is actually might be one reason to define __all__
When importing a module, you might want that doing from mod import * will import only a minimal amount of things. Even if you prefix everything correctly, there could be reasons not to import everything.
The other problem that I had once was defining a gettext shortcut. The translation function was _ which would not get imported. Even if it is prefixed "in theory" I still wanted it to get exported.
One other reason as stated above is importing a module that imports a lot of thing. As python cannot make the difference between symbols created by imports and the one defined in the actual module. It will automatically reexport everything that can be reexported. For that reason, it could be wise to explicitely limit the thing exported to the things you want to export.
With that in mind, you might want to have some prefixed symbols exported by default. Usually, you don't redefine __all__. Whenever you need it to do something unusual then it may make sense to do it.
I want to make something like plugin system but can't make it working. To be specific I have some requirements.
I have main script who should search for other python scripts in ./plugins dir and load them.
This main script is searching for classes who inherits from Base using globals()
If I place these classes in the same main file it works very well but I can't get it worked as I want.
Is it possible to do this in Python?
I try to make some like this:
source: plugins/test.py
class SomeClass(Base):
def __init__(self):
self.name = "Name of plugin"
Main script just execute some methods on this class.
You could either import the python file dynamically or use the exec statement (make sure to define a context to execute in, otherwise the context you use the statement in will be used). Then use Base.__subclasses__, assuming Base being a new-style class, or call a function from the imported plugin module. In the latter case, you must provide a plugin-registration mechanism.
Use http://docs.python.org/2/library/imp.html#imp.load_module
For py3 I think there is importlib but I don't know how to use that one offhand.
Try importing the modules using imp -- imp.loadmodule will let you create namespace names dynamically if you need to. Then you can use inspect.getmembers() and inspect.is_class() to find the classes in your imported module (example code in this answer) to find all the clases defined there. Test those for being subclasses of your plugin.
...or, more pythonically, just use hasattr to find out if the imported classes 'quack like a duck' (ie, have the methods you expect from your plugin).
PS - I'm assuming you're asking for python 2.x. Good idea to tag the post with version # in future.
The package I am documenting consists of a set of *.py files, most containing one class with a couple of files being genuine modules with functions defined. I do not need to expose the fact that each class is in a module so I have added suitable from statements in the __init__.py file e.g.
from base import Base
so that the user can use the import pkg command and does not then have to specify the module that contains the class:
import pkg
class MyBase(pkg.Base): # instead of pkg.base.Base ...
...
The problem is that Sphinx insists on documenting the class as pkg.base.Base. I have tried to set the add_module_names = False in conf.py. However this results in Sphinx showing the class as simply Base instead of pkg.Base. Additionally this also ruins the documentation of the couple of *.py files that are modules.
How do I make Sphinx show a class as pkg.Base?
And how do I set the add_module_names directive selectively for each *.py file?
Here is a way to accomplish what the OP asks for:
Add an __all__ list in pkg/__init__.py:
from base import Base # Or use 'from base import *'
__all__ = ["Base"]
Use .. automodule:: pkg in the .rst file.
Sphinx will now output documentation where the class name is shown as pkg.Base instead of pkg.base.Base.
I've incorporated the answers I found in a scalable-ish form factor:
my_project/
__init__.py
mess.py
mess.py:
class MyClass:
pass
class MyOtherClass(MyClass):
pass
__init__.py:
from .mess import MyClass, MyOtherClass
__all_exports = [MyClass, MyOtherClass]
for e in __all_exports:
e.__module__ = __name__
__all__ = [e.__name__ for e in __all_exports]
This seems to have worked pretty well for me.
I would like to provide a more generalized approach.
The variable __all__ is filled up based on dir(). But the sub-packages name (here mypackage) and all in-build attributes (starting with __) are ignored.
from .mypackage import *
__all__ = []
for v in dir():
if not v.startswith('__') and v != 'mypackage':
__all__.append(v)
Short answer: You shouldn't. Just point the sphinx to the directory of your code. Sphinx documents the code and shows the module hirarchy. How the module finally will be imported is purely in the hand of the developer, but not a responsibility of the documentation tool.
Let's assume that we have a system of modules that exists only on production stage. At the moment of testing these modules do not exist. But still I would like to write tests for the code that uses those modules. Let's also assume that I know how to mock all the necessary objects from those modules. The question is: how do I conveniently add module stubs into current hierarchy?
Here is a small example. The functionality I want to test is placed in a file called actual.py:
actual.py:
def coolfunc():
from level1.level2.level3_1 import thing1
from level1.level2.level3_2 import thing2
do_something(thing1)
do_something_else(thing2)
In my test suite I already have everything I need: I have thing1_mock and thing2_mock. Also I have a testing function. What I need is to add level1.level2... into current module system. Like this:
tests.py
import sys
import actual
class SomeTestCase(TestCase):
thing1_mock = mock1()
thing2_mock = mock2()
def setUp(self):
sys.modules['level1'] = what should I do here?
#patch('level1.level2.level3_1.thing1', thing1_mock)
#patch('level1.level2.level3_1.thing1', thing2_mock)
def test_some_case(self):
actual.coolfunc()
I know that I can substitute sys.modules['level1'] with an object containing another object and so on. But it seems like a lot of code for me. I assume that there must be much simpler and prettier solution. I just cannot find it.
So, no one helped me with my problem and I decided to solve it by myself. Here is a micro-lib called surrogate which allows one to create stubs for non-existing modules.
Lib can be used with mock like this:
from surrogate import surrogate
from mock import patch
#surrogate('this.module.doesnt.exist')
#patch('this.module.doesnt.exist', whatever)
def test_something():
from this.module.doesnt import exist
do_something()
Firstly #surrogate decorator creates stubs for non-existing modules, then #patch decorator can alter them. Just as #patch, #surrogate decorators can be used "in plural", thus stubbing more than one module path. All stubs exist only at the lifetime of decorated function.
If anyone gets any use of this lib, that would be great :)