Develop Reference

Multi layer package in python

I have a python package with packages in it. This explanation seems strange, so I'll include my package's structure:
package\_
__init__.py
subpackage1\_
__init__.py
file1.py
subpackage2\_
__init__.py
file2.py
(I'm simplifying it for easier understanding).
The __init__.py on the top level looks like this:
__all__ = ["subpackage1", "subpackage2"]
And, for some reason, when importing the package, it dosen't recognise anythong from file1.py or file2.py. Any ideas how to fix it?
If you need more details, here's the project on github: https://github.com/Retr0MrWave/mathModule
. The directory I called package is mathmodule_pkg in the actual project

Filling the __all__ field with names does not make imports possible, it merely serves as a hint of what you mean to make importable. This hint is picked up by star-imports to restrict what is imported, and IDEs like pycharm also use it to get an idea of what is and isn't exposed - but that's about it.
If you want to enable top-level imports of your nested classes and functions, you need to
import them into the top-level __init__.py
bind them to names that can be used for the import
optionally, reference said names in __all__ to make the API nice and obvious
Using the project you're referencing as an example, this is what it would look like:
mathmodule_pkg/__init__.py
import mathmodule_pkg.calculus.DerrivativeAndIntegral #1
integral = mathmodule_pkg.calculus.DerrivativeAndIntegral.integral #2
__all__ = ['integral'] # 3
Using the very common form of from some.package import some_name we can combine steps 1 and 2 and reduce the potential for bugs when re-binding the name:
from mathmodule_pkg.calculus.DerrivativeAndIntegral import integral # 1 and 2
__all__ = ['integral'] # 3
Using either form, after installing your package the following will be possible:
>>> from mathmodule_pkg import integral
>>> integral(...)

How to exclude imports from automodapi output?

I am trying to use automodapi to generate documentation for my Django project. When I call automodapi like this:
.. automodapi:: mypackage.mymodule
the output includes all imported classes and functions, e.g, the Django Model class, in the index of Functions and Classes. I would like to exclude the imports and only list those classes and functions declared in the module I specified.
I couldn't see anything about this in the documentation.
Is there a way to do this, preferably without modifying modules?
UPDATE: #saimn has provided a working solution using __all__ but my project doesn't use __all__. It would be nice if there was a solution that didn't involve modifying the modules.

Patching automodapi to only include locals also addresses this issue while not requiring any changes to your code:
def patch_automodapi(app):
"""Monkey-patch the automodapi extension to exclude imported members"""
from sphinx_automodapi import automodsumm
from sphinx_automodapi.utils import find_mod_objs
automodsumm.find_mod_objs = lambda *args: find_mod_objs(args[0], onlylocals=True)
def setup(app):
app.connect("builder-inited", patch_automodapi)
Source: https://github.com/astropy/sphinx-automodapi/issues/119
The above snippet goes into your conf.py sphinx configuration file.

You can use the __all__ variable (this should probably be stated more clearly in the documentation).

Make methods available at higher level in a package

Consider the following package structure:
foo/ # package name
spam/ # module name
__init__.py
eggs.py # contains "bar" method
exceptions.py # contains "BarException" class
Now in order to call the bar method, we have to do
import spam
spam.eggs.bar()
I'd like to lose eggs.
Now I know it is possible to import ... as (and from ... import), but is there no way to make methods available higher up in a tree?
Things I do not want to resort to:
lots of from ... import ...
putting my eggs.py code in __init__.py instead
starred imports
long names like spam.exceptions.BarException (possibly longer)
An example would be to have exceptions.py where I define my exception classes.
Whenever I would want to make them available to a user I wouldn't want them to use spam.exceptions.BarException, but rather be able to use spam.BarException.
Goal:
import spam
try:
spam.bar() # in this case throws BarException
except spam.BarException:
pass

Note that, contrary to your comments, the top foo is not the package name, it's just a directory that's (presumably) on your sys.path somewhere, and spam is not the module name but the package name, and eggs is the module name. So:
foo/ # directory package is in
spam/ # package name
__init__.py
eggs.py # contains "bar" method
exceptions.py # contains "BarException" class
The key to what you want to do is this:
Any global names in spam/__init__.py are members of the spam package. It doesn't matter whether they were actually defined in __init__.py, or imported from somewhere else.
So, if you want to make the spam.eggs.bar function available as spam.bar, all you have to do is add this line to spam/__init__.py:
from .eggs import bar
If you have an __all__ attribute in spam/__init__.py to define the public attributes of spam, you will want to add bar to that list:
__all__ = ['other', 'stuff', 'directly', 'in', 'spam', 'bar']
If you want to re-export everything public from spam.eggs as a public part of spam, you can just do this:
from .eggs import *
__all__ = ['other', 'stuff', directly', 'in', spam'] + eggs.__all__
And of course you can extend this to more than one child module:
from .eggs import *
from .exceptions import *
__all__ = (['other', 'stuff', directly', 'in', spam'] +
eggs.__all__ +
exceptions.__all__)
This is common in the stdlib, and in popular third-party packages. For a good example, see the source to asyncio/__init__.py from Python 3.4.
However, it's only really common in this exact case: you want your uses to be able to treat your package as if it were a simple, flat module, but it actually has some internal structure (either because the implementation would be too complicated otherwise, or because occasionally users will need that structure). If you're pulling in names from a grandchild, sibling, or parent instead of a child, you're probably abusing the idiom (or at least you should stop and convince yourself that you're not).

In your __init__.py, you can import things from other modules in the package. If in __init__.py you do from .eggs import bar, then someone can do import spam and access spam.bar. If in __init__.py you do from .exceptions import BarException, then someone can do import spam and then do spam.BarException.
However, you should be wary of going too far with this. Using nesting in packages and modules has a purpose, namely to create separate namespaces. Explicitly importing a few common things from a submodule to the top level is fine, but if you start trying to implicitly make everything available at the top level, you set yourself up for name collisions down the road (e.g., if one module defines something called Blah and then later another module also does so, without realizing they will collide when they're both imported to the top level).
"Forcing" users to use from is not an onerous requirement. If cumbersome imports are required to use your library, that may be a sign that your package/module structure is too cumbersome, and you should combine some things rather than splitting them up into separate directories/files.
Incidentally, the file structure you have indicated in your post has some problems. The top-level foo as you have shown is not a package, since it doesn't have an __init__.py. The second-level spam is not a module, since it is not a file. In your example, spam is a package, and it has inside it a module called eggs (in the file eggs.py); the top-level foo directory has no status in the Python packaging system.

How do I document classes imported from other modules - hence without the declaration module's name?

The package I am documenting consists of a set of *.py files, most containing one class with a couple of files being genuine modules with functions defined. I do not need to expose the fact that each class is in a module so I have added suitable from statements in the __init__.py file e.g.
from base import Base
so that the user can use the import pkg command and does not then have to specify the module that contains the class:
import pkg
class MyBase(pkg.Base): # instead of pkg.base.Base ...
...
The problem is that Sphinx insists on documenting the class as pkg.base.Base. I have tried to set the add_module_names = False in conf.py. However this results in Sphinx showing the class as simply Base instead of pkg.Base. Additionally this also ruins the documentation of the couple of *.py files that are modules.
How do I make Sphinx show a class as pkg.Base?
And how do I set the add_module_names directive selectively for each *.py file?

Here is a way to accomplish what the OP asks for:
Add an __all__ list in pkg/__init__.py:
from base import Base # Or use 'from base import *'
__all__ = ["Base"]
Use .. automodule:: pkg in the .rst file.
Sphinx will now output documentation where the class name is shown as pkg.Base instead of pkg.base.Base.

I've incorporated the answers I found in a scalable-ish form factor:
my_project/
__init__.py
mess.py
mess.py:
class MyClass:
pass
class MyOtherClass(MyClass):
pass
__init__.py:
from .mess import MyClass, MyOtherClass
__all_exports = [MyClass, MyOtherClass]
for e in __all_exports:
e.__module__ = __name__
__all__ = [e.__name__ for e in __all_exports]
This seems to have worked pretty well for me.

I would like to provide a more generalized approach.
The variable __all__ is filled up based on dir(). But the sub-packages name (here mypackage) and all in-build attributes (starting with __) are ignored.
from .mypackage import *
__all__ = []
for v in dir():
if not v.startswith('__') and v != 'mypackage':
__all__.append(v)

Short answer: You shouldn't. Just point the sphinx to the directory of your code. Sphinx documents the code and shows the module hirarchy. How the module finally will be imported is purely in the hand of the developer, but not a responsibility of the documentation tool.

python: list modules within the package

I have a package with a few modules, each module has a class (or a few classes) defined within it. I need to get the list of all modules within the package. Is there an API for this in python?
Here is the file structure:
\pkg\
\pkg\__init__.py
\pkg\module1.py -> defines Class1
\pkg\module2.py -> defines Class2
\pkg\module3.py -> defines Class3 and Class31
from within module1 I need to get the list of modules within pkg, and then import all the classes defined in these modules
Update 1:
Ok, after considering the answers and comments below I figured, that it's not that's easy to achieve my need. In order to have the code I proposed below working, all the modules should be explicitly imported beforehand.
So now the new concept:
How to get the list of modules within a package without loading the modules? Using python API, i.e. - without listing all the files in the package folder?
Thanks
ak

One ad-hoc approach is list the files in the directory, import each file dynamically with __import__ and then list the classes of the resulting module.

ok, this was actually pretty straightforward:
import pkg
sub_modules = (
pkg.__dict__.get(a) for a in dir(pkg)
if isinstance(
pkg.__dict__.get(a), types.ModuleType
)
)
for m in sub_modules:
for c in (
m.__dict__.get(a) for a in dir(m)
if isinstance(m.__dict__.get(a), type(Base))
):
""" c is what I needed """