I cannot eliminate the doctest flags (ie. <BLANKLINE>, # doctest: +ELLIPSIS) for the html output. I am able to generate the documentation as I would like, so no errors there but it includes theses flags which I would like removed. Sphinx documentation here claims this is possible so I must be doing something wrong. My documentation examples are in numpy style and I have tried using both the napoleon and numpydoc extensions.
Here are the steps I have taken.
run sphinx-quickstart (enabling autodoc and doctest extensions)
run sphinx-apidoc to generate .rst files
run make doctest (all tests are passing)
run make html
I have tried the setting trim_doctest_flags and doctest_test_doctest_blocks variables in conf.py with no success.
Is there something I am missing to trigger sphinx to remove these for the html docs? I am hoping this is enough information to get pointed in the right direction since the docs look good except for this one issue. However, I can provide more details or an example if necessary.
Update: MCV Example (Using Sphinx 1.8.2)
directory and file structure
.
├── trial
│ ├── __init__.py
│ └── trial.py
└── trialDocs
├── build
├── Makefile
└── source
├── _static
├── _templates
├── conf.py
├── index.rst
├── modules.rst
└── trial.rst
conf.py
# -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
import os
import sys
sys.path.insert(0, os.path.abspath('../../trial'))
project = 'trial'
copyright = '2019, trial'
author = 'trial'
version = ''
release = 'trial'
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.doctest',
'sphinx.ext.napoleon',
]
templates_path = ['_templates']
source_suffix = '.rst'
master_doc = 'index'
language = None
exclude_patterns = []
pygments_style = None
html_theme = 'alabaster'
htmlhelp_basename = 'trialdoc'
latex_elements = {}
latex_documents = [(master_doc, 'trial.tex', 'trial Documentation', 'trial', 'manual'),]
man_pages = [(master_doc, 'trial', 'trial Documentation', [author], 1)]
texinfo_documents = [(master_doc, 'trial', 'trial Documentation', author, 'trial', 'One line description of project.', 'Miscellaneous'),]
epub_title = project
epub_exclude_files = ['search.html']
doctest_global_setup = """
from trial import *
"""
trim_doctest_flags=True
trial.rst - this was generated using sphinx-apidoc
trial package
=============
Module contents
---------------
.. automodule:: trial
:members:
:undoc-members:
:show-inheritance:
trial.py
def withBlankline():
"""
Use blanklines in example.
Determine if sphinx will eliminate <BLANKLINE> for html.
Examples
--------
>>> withBlankline()
<BLANKLINE>
blanklines above and below
<BLANKLINE>
"""
print()
print('blanklines above and below')
print()
class Example():
def __init__(self):
pass
def withEllipsis(self):
"""
Use ellipsis in example.
Determine if sphinx will eliminate # doctest: +ELLIPSIS for html.
Examples
--------
>>> e = Example()
>>> e.withEllipsis() # doctest: +ELLIPSIS
abc...xyz
"""
print('abcdefghijklmnopqrstuvwxyz')
using make html or sphinx-build -b html source build
trial.html output:
Based on the comment by #mzjn, it appears that this was a bug, fixed in Sphinx 2.2.0:
Issue: doctest comments not getting trimmed since Sphinx 1.8.0 - Issue #6545
Related
I generate the documentation of my Python code from the docstrings via Doxygen (1.9.1) in addition with doxypypy (git version from today).
My project is called Project and the packages name in it (which should be imported) is mypackage. When I look into the tree sidebar of the generated html it looks like this:
└── Project
└── Packages
└── Packages
└──mypackages
The two packages are linking to the same target: ../html/namespaces.html.
The files and folders are structured like this
Project
├── LICENSE
├── README.md
├── docs
└── ...
└── src
├── mypackage
│ ├── a.py
│ ├── __init__.py
│ └── _mypackage.py
├── setup.cfg
└── setup.py
The Doxyfile is located in Project/docs, doxygen is run in there and use ../src/mypackage as INPUT directory.
More details
__init__.py
__version__ = '0.0.1a'
from ._mypackage import *
_mypackage.py
# -*- coding: utf-8 -*-
"""Example __init__.py short.
Now some longer with multiple lines. Here
comes the scond line.
"""
def foo(bar):
"""
This is foo() in mypackage.
Args:
bar (str): A paramenter.
Returns:
(int): Fixed seven.
"""
print(bar)
return 7
a.py
# -*- coding: utf-8 -*-
"""This is mypackage.a
"""
import mypackage
def bar(bar):
"""
This is the function named bar.
The function calls `mypackage.foo()` and returns an 'a'.
Paramters:
bar (str): Just a parameter.
Returns:
str: Just an 'a'.
"""
mypackage.foo(bar)
return('a')
Some (maybe) related Doxyfile settings
BRIEF_MEMBER_DESC = YES
REPEAT_BRIEF = YES
ALWAYS_DETAILED_SEC = NO
FULL_PATH_NAMES = YES
JAVADOC_AUTOBRIEF = NO
PYTHON_DOCSTRING = YES
OPTIMIZE_OUTPUT_FOR_C = NO
OPTIMIZE_OUTPUT_JAVA = YES
OPTIMIZE_FOR_FORTRAN = NO
OPTIMIZE_OUTPUT_VHDL = NO
OPTIMIZE_OUTPUT_SLICE = NO
MARKDOWN_SUPPORT = YES
EXTRACT_ALL = YES
EXTRACT_PRIVATE = YES
EXTRACT_PACKAGE = YES
EXTRACT_STATIC = YES
EXTRACT_LOCAL_CLASSES = YES
EXTRACT_LOCAL_METHODS = YES
EXTRACT_ANON_NSPACES = NO
RESOLVE_UNNAMED_PARAMS = YES
HIDE_UNDOC_MEMBERS = NO
HIDE_UNDOC_CLASSES = NO
HIDE_FRIEND_COMPOUNDS = NO
HIDE_IN_BODY_DOCS = NO
INPUT = ../src/mypackage
FILE_PATTERNS =
RECURSIVE = YES
FILTER_PATTERNS = *.py=./py_filter
GENERATE_HTML = YES
GENERATE_TREEVIEW = YES
I opened an Issue about that.
I am trying to set up Bazel on an existing project that consists of three applications in Python and Groovy, and a shared protobuf IDL.
For the Python applications, I currently have a custom command in setup.py that generates a Python module from the protobuf IDL. When generating the python module, I place it inside the application packages, so it can be imported like any other module in the application.
When trying to put the whole project under Bazel I'm struggeling to find out how to deal with the generated python module. protoc will only generate a single file. In order to put the file in a package, I need to create a directory structure and move the file into place. Some googling has lead me to a solution that combines some pkg_tar rules to create a tarball with the correct layout, but I can't figure out how to make the jump to making this into a python library.
The files are laid out like this:
.
├── BUILD
├── protobuf
│ └── messages.proto
└── python
└── ibidem
├── __init__.py
└── codetanks
├── __init__.py
└── domain
└── __init__.py
I want the generated module to be placed in ibidem/codetanks/domain, so that it can be imported with from ibidem.codetanks.domain import messages_pb2.
My current BUILD file:
load("#build_stack_rules_proto//python:python_proto_library.bzl", "python_proto_library")
load("#rules_pkg//:pkg.bzl", "pkg_tar", "pkg_deb")
proto_library(
name = "messages_proto",
srcs = ["protobuf/messages.proto"],
)
python_proto_library(
name = "messages_python_proto",
deps = [":messages_proto"],
)
pkg_tar(
name = "python_messages_tarball",
strip_prefix = "protobuf/",
package_dir = "ibidem/codetanks/domain",
srcs = [":messages_python_proto"],
)
filegroup(
name = "python_domain_files",
srcs = glob([
"python/**/*.py",
]),
)
pkg_tar(
name = "python_domain_tarball",
strip_prefix = "python/",
srcs = [":python_domain_files"],
)
# This fails because the tarballs doesn't have the `py` or `PyInfo` provider .
# If I use a `pkg_tar` rule here, the tarball has exactly the contents I'd want to have as a python library.
py_library(
name = "python",
deps = [
":python_domain_tarball",
":python_messages_tarball",
],
)
I've found some places that say that this can be solved by putting the messages.proto file inside the directory structure in the same place as I want the generated file to wind up. That sounds like a bad workaround, considering that placement won't make any sense for any other language than Python. I also generate a java package, and in the future the plan is to add other languages too.
Is this simply a limitation of Bazel, or can it be solved in some fancy way that I haven't been able to google my way to?
With the link provided by Sjoerd Visscher, I found a solution that seems to do the trick. First step was to separate into language specific packages. Then use copy_file to move the generated file into a subdirectory. Once that was done, combining the helper files with the generated file in a py_library was pretty straight forward.
The file layout is now:
.
├── protobuf
│ ├── BUILD
│ └── messages.proto
└── python
├── BUILD
└── ibidem
├── __init__.py
└── codetanks
├── __init__.py
└── domain
└── __init__.py
Contents of protobuf/BUILD:
package(default_visibility=["//domain:__subpackages__"])
load("#build_stack_rules_proto//python:python_proto_library.bzl", "python_proto_library")
proto_library(
name = "messages_proto",
srcs = ["messages.proto"],
)
python_proto_library(
name = "python_messages_proto",
deps = [":messages_proto"],
)
Contents of python/BUILD:
load("#build_stack_rules_proto//python:python_proto_library.bzl", "python_proto_library")
load("#rules_python//python:defs.bzl", "py_library")
load("#bazel_skylib//rules:copy_file.bzl", "copy_file")
copy_file(
name="python_messages_file",
src="//domain/protobuf:python_messages_proto",
out="ibidem/codetanks/domain/messages_pb2.py",
)
filegroup(
name = "python_helper_files",
srcs = glob([
"ibidem/**/__init__.py",
]),
)
py_library(
name = "messages",
srcs = [
":python_helper_files",
":python_messages_file",
],
visibility = ["//visibility:public"]
)
I have a dir which has such structure:
├── aaa.py
├── src
│ └── subsrc
│ ├── else.py
│ └── util.py (there is a "foo" function")
└── tests
├── __init__.py
└── unittests
├── __init__.py
└── test_aaa.py
so "aaa.py", "tests" dir and "src" dir are in project root. and in "test_aaa.py", I use mock to mock function in "util.py":
from src.subsrc.util import foo
import pytest
from unittest import mock
#mock.patch("src.subsrc.util.foo")
def test_foo(mock):
mock.return_value = 111
and then I run python3.7 -m pytest inside "unittests" dir, it worked. This makes sense to me since pytest will find the first dir without __init__.py and then add it to PATH(in this case project root dir will be added) so it could find "src.subsrc.util.foo".
But then I made a small change to "test_aaa.py", in its "mock.patch", I added "aaa" at the beginning:
from src.subsrc.util import foo
import pytest
from unittest import mock
#mock.patch("aaa.src.subsrc.util.foo")
def test_foo(mock):
mock.return_value = 111
it still worked, "aaa.py" is an executable, in "aaa.py":
#!python3.7
from src.subsrc.else import other
if __name__ = "__main__":
# ...
pass
I am very confused why #mock.patch("aaa.src.subsrc.util.foo") also worked, is Python so smart that it could ignore 'aaa' then go "src.subsrc.." to find what it needs? Thanks!
update:
I suspect if because "aaa.py"'s name is special so I changed it to different names, but it still worked. Like I change it to "bbb.py", then in mock.patch, "aaa.src..." does not work but "bbb.src..." still worked. So I am sure "mock.patch" find this executable first.
update:
I guess it could be related to how "mock.patch()" works?
Your example seems to be a bit too stripped-down, but I'll try to expand it in order to explain. When reading about mocks in Python, you will often encounter the phrase "mock it where it's used", which isn't really helpful if you are new to the topic (but here's an excellent article on this concept).
In your test_aaa.py you will probably want to test some functionality of your aaa.py module, which may call some function from src/subsrc/util.py. After importing your foo() function in the aaa.py module, that's the exact location where you should point #mock.patch to: #mock.patch("aaa.foo"). By doing this, your mock will have access to all invocations of foo() in the functions you are about to test, namely aaa.do_something(). I've expanded your example as follows:
# aaa.py
from src.subsrc.util import foo
def do_something():
return foo()
if __name__ == "__main__":
value = do_something()
print(f"value is {value}")
# src/subsrc/util.py
def foo():
return 222
# tests/unittests/test_aaa.py
from unittest import mock
from aaa import do_something
#mock.patch("aaa.foo")
def test_foo(foo_mocked):
foo_mocked.return_value = 111
value = do_something()
assert value == 111
When executing this like python aaa.py, I get the output as expected (value is 222) while the test passes with its assert value == 111.
In your example, #mock.patch("src.subsrc.util.foo") obviously worked, but probably didn't do what you intended. From your example code, I cannot see how #mock.patch("aaa.src.subsrc.util.foo") shouldn't have returned a ModuleNotFoundError.
Suppose we have a project like this:
project-path
├── root
│ ├── BUILD
│ ├── gen
│ │ ├── a2.txt
│ │ └── a.txt
│ └── use.py
└── WORKSPACE
And in use.py:
f = open("gen/a.txt", "r")
f2 = open("gen/a2.txt", "r")
print(f.read())
print(f2.read())
And BUILD:
py_binary(
name = "use",
srcs = ["use.py"],
data = ["gen/a.txt", "gen/a2.txt"],
)
when I bazel run root:use, it errors:
FileNotFoundError: [Errno 2] No such file or directory: 'gen/a.txt'
It expects paths relative to the WORKSPACE directory, not the current package (root/gen/a.txt here). But I want to access files relative to each package.
Add Skylib to your project, i.e. extend you WORKSPACE file:
WORKSPACE
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "bazel_skylib",
sha256 = "74d544d96f4a5bb630d465ca8bbcfe231e3594e5aae57e1edbf17a6eb3ca2506",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.3.0/bazel-skylib-1.3.0.tar.gz",
"https://github.com/bazelbuild/bazel-skylib/releases/download/1.3.0/bazel-skylib-1.3.0.tar.gz",
],
)
load("#bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
bazel_skylib_workspace()
Replace the path in the bash script using text replacement:
load("#bazel_skylib//rules:expand_template.bzl", "expand_template")
expand_template(
name = "modifiy_use_for_bazel",
out = "prepared_for_bazel_use.py",
substitutions = {
"gen/a.txt": "root/gen/a.txt",
"gen/a2.txt": "root/gen/a.txt",
},
template = "use.py",
)
py_binary(
name = "use",
main = "prepared_for_bazel_use.py",
srcs = ["prepared_for_bazel_use.py"],
data = [
"gen/a.txt",
"gen/a2.txt",
],
)
You can run now the script via bazel run root:use
Note: Tested with Bazel 6.0.0
To make this all a bit more convenient to use you could implement your own rule for it since this seems to be a common problem (e.g. when supporting two build systems at the same time where Bazel is not the primary build system and the other build system can cope with relative path names.)
A similar problem is described here: Change test execution directory in Bazel?
Suppose in this example, how to access the respective config.json file in conftest fixtures upon executing test-suite using pytest.
$ pwd
/home/user/repo/main
$ pytest testcases/project_(1/2)/test_suite_(1/2).py
Directory structure:
├── main
│ ├── conftest.py # conftest file for my fixtures
│ ├── testcases
│ ├── project_1
│ │ (contains these files -- test_suite_1.py, config.json)
│ └── project_2
│ (contains these files -- test_suite_2.py, config.json)
├── workflows
│ └── libs
You can access the path of the currently executed module via request.node.fspath and build the path to the config.json relative to it. request is a fixture provided by pytest. Here's an example based on the directory structure you provided.
# main/conftest.py
import json
import pathlib
import pytest
#pytest.fixture(autouse=True)
def read_config(request):
file = pathlib.Path(request.node.fspath)
print('current test file:', file)
config = file.with_name('config.json')
print('current config file:', config)
with config.open() as fp:
contents = json.load(fp)
print('config contents:', contents)
If you copy the code above to your conftest.py and run the tests with -s, you should get an output similar to this:
$ pytest -sv
=============================== test session starts ===============================
platform linux -- Python 3.6.5, pytest-3.4.1, py-1.5.3, pluggy-0.6.0 -- /data/gentoo64/usr/bin/python3.6
cachedir: .pytest_cache
rootdir: /data/gentoo64/tmp/so-50329629, inifile:
collected 2 items
main/project1/test_one.py::test_spam
current file: /data/gentoo64/tmp/so-50329629/main/project1/test_one.py
current config: /data/gentoo64/tmp/so-50329629/main/project1/config.json
config contents: {'name': 'spam'}
PASSED
main/project2/test_two.py::test_eggs
current file: /data/gentoo64/tmp/so-50329629/main/project2/test_two.py
current config: /data/gentoo64/tmp/so-50329629/main/project2/config.json
config contents: {'name': 'eggs'}
PASSED
============================= 2 passed in 0.08 seconds ============================
Use parsed config values
You can access the parsed JSON data by returning it in the fixture and using the fixture as one of the test arguments. I slightly modified the fixture from above so it returns the parsed data and removed the autouse=True:
#pytest.fixture
def json_config(request):
file = pathlib.Path(request.node.fspath.strpath)
config = file.with_name('config.json')
with config.open() as fp:
return json.load(fp)
Now simply use the fixture name in the test arguments, the value will be what the fixture returns. for example:
def test_config_has_foo_set_to_bar(json_config):
assert json_config['foo'] == 'bar'