Develop Reference

How to get a list of warnings from sphinx compilation

I am developing a sphinx based collaborative writing tool. Users access the web application (developed in python/Flask) to write a book in sphinx and compile it to pdf.
I have learned that in order to compile a sphinx documentation from within python I should use
import sphinx
result = sphinx.build_main(['-c', 'path/to/conf',
'path/to/source/', 'path/to/out'])
So far so good.
Now my users want the app to show them their syntax mistakes. But the output (result in the example above) only gives me the exit code.
So, how do I get a list of warnings from the build process?
Perhaps I am being too ambitious, but since sphinx is a python tool, I was expecting to have a nice pythonic interface with the tool. For example, the output of sphinx.build_main could be a very rich object with warnings, line numbers...
On a related note, the argument to the method sphinx.build_main looks just like a wrapper to the command line interface.

sphinx.build_main() calls sphinx.cmdline.main(), which in turn creates a sphinx.application.Sphinx object. You could create such an object directly (instead of "making system calls within python"). Use something like this:
import os
from sphinx.application import Sphinx
# Main arguments
srcdir = "/path/to/source"
confdir = srcdir
builddir = os.path.join(srcdir, "_build")
doctreedir = os.path.join(builddir, "doctrees")
builder = "html"
# Write warning messages to a file (instead of stderr)
warning = open("/path/to/warnings.txt", "w")
# Create the Sphinx application object
app = Sphinx(srcdir, confdir, builddir, doctreedir, builder,
warning=warning)
# Run the build
app.build()

Assuming you used sphinx-quickstart to generate your initial Sphinx documentation set with a makefile, then you can use make to build docs, which in turn uses the Sphinx tool sphinx-build. You can pass the -w <file> option to sphinx-build to write warnings and errors to a file as well as stderr.
Note that options passed through the command line override any other options set in the makefile and conf.py.

In setup.py, how to test if PyCapsule_New is defined?

setup.py has a feature for testing if functions are defined:
compiler = distutils.ccompiler.new_compiler ()
if compiler.has_function ('foo_new', libraries=("foo",)):
define_macros.append (('HAVE_FOO_NEW', '1'))
However I can't seem to use this for Python extension functions (specifically PyCapsule_New). The following does not define anything:
if compiler.has_function ('PyCapsule_New'):
define_macros.append (('HAVE_PYCAPSULE_NEW', '1'))
I seem to need to put something in the libraries argument, but what? The name of the Python library changes, and is not available in distutils.sysconfig except as a gcc parameter (eg. BLDLIBRARY is defined as something like -L. -lpython2.7).
It seems like such an obvious/common thing to want to do so the code will work on multiple versions of Python, but no setup.py scripts I can find use has_function in this way.

Instead of doing configure checks for Python features, you could do some compile-time testing. Ideally you could check against the Python version (PY_MAJOR_VERSION, PY_MINOR_VERSION), but you could also rely on macros defined inside the headers.
For your specific feature, note that the Py_CAPSULE_H macro is defined once the header pycapsule.h is included (via Python.h).

Examples of entry_point usage

I discoverd entry_points of setuptools:
http://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins
quote: setuptools supports creating libraries that “plug in” to extensible applications and frameworks, by letting you register “entry points” in your project that can be imported by the application or framework.
But I have not seen a project using them.
Are there examples of projects which use them?
If not, why are they not used?

There are loads of examples. Any project that defines console scripts uses them, for example. A quick search on GitHub gives you plenty to browse through.
I'll focus on one specific example (one that is not on GitHub): Babel.
Babel uses both entry_points for both console scripts and to define extension points for translatable text extraction. See their setup.py source:
if have_setuptools:
extra_arguments = dict(
zip_safe = False,
test_suite = 'babel.tests.suite',
tests_require = ['pytz'],
entry_points = """
[console_scripts]
pybabel = babel.messages.frontend:main
[distutils.commands]
compile_catalog = babel.messages.frontend:compile_catalog
extract_messages = babel.messages.frontend:extract_messages
init_catalog = babel.messages.frontend:init_catalog
update_catalog = babel.messages.frontend:update_catalog
[distutils.setup_keywords]
message_extractors = babel.messages.frontend:check_message_extractors
[babel.checkers]
num_plurals = babel.messages.checkers:num_plurals
python_format = babel.messages.checkers:python_format
[babel.extractors]
ignore = babel.messages.extract:extract_nothing
python = babel.messages.extract:extract_python
javascript = babel.messages.extract:extract_javascript
""",
)
Tools like pip and zc.buildout use the console_scripts entry point to create commandline scripts (one called pybabel, running the main() callable in the babel.messages.frontend module).
The distutils.commands entry points defines additional commands you can use when running setup.py; these can be used in your own projects to invoke Babel command-line utilities right from your setup script.
Last, but not least, it registers its own checkers and extractors. The babel.extractors entry point is loaded by the babel.messages.extract.extract function, using the setuptools pkg_resources module, giving access to all installed Python projects that registered that entry point. The following code looks for a specific extractor in those entries:
try:
from pkg_resources import working_set
except ImportError:
pass
else:
for entry_point in working_set.iter_entry_points(GROUP_NAME,
method):
func = entry_point.load(require=True)
break
This lets any project register additional extractors; simply add an entry point in your setup.py and Babel can make use of it.

Sentry is a good example. Sentry's author even created a django package named Logan to convert standard django management commands to console scripts.

how to set bug tracker url in setup.py script

I have just discovered the pypi web UI have a field 'Bug tracker URL' in edit of egg metadata.
This field exists so I guess it is supported in setup.py but I can't find anything about this using google.
So the question how do I set up this field in my setup.py so when doing a dist release on pypi it can be automaticly filled.

The entry is called bugtrack_url, but it's not being picked up from setup.py.
From context and code I understand it was intended to be used through-the-web on PyPI only, as per-project metadata, and not the usual per-release information.
The field is now considered a legacy field (hardcoded to None) and you instead add such information through the Project-URL list, which you can set in setuptools via the project_urls entry:
project_urls={
'Documentation': 'https://packaging.python.org/tutorials/distributing-packages/',
'Funding': 'https://donate.pypi.org',
'Say Thanks!': 'http://saythanks.io/to/example',
'Source': 'https://github.com/pypa/sampleproject/',
'Tracker': 'https://github.com/pypa/sampleproject/issues',
},
This option was finally added to setuptools in November 2017, and landed in version 38.3.0.

Bug tracker URL on PyPi project
In setup.py use project_urls in the setup :
setup(
...
project_urls={
'Documentation': 'https://readthedocs.io/',
'Funding': 'https://donate.pypi.org',
'Say Thanks!': 'http://saythanks.io/to/example',
'Source': 'https://github.com/pypa/sampleproject/',
'Tracker': 'https://github.com/pypa/sampleproject/issues',
},
...
)
The dict order is kept but listed in reversed on PyPi:
About PyPi bugtracker_url legacy code
pypa/warehouse Issue #233
bugtrack_url: IIRC it was something added by the PyPI maintainers to help projects, but in parallel PEP 345 introduced Project-URL which was intended to cover source code repository, bug tracker, mailing list, etc. If PEP 426 or one of its companion keeps Project-URL (and maybe improves it with defined labels for common sites, e.g. "repository"), then this special case becomes redundant.
And
At the moment, it looks like this is hardcoded to None in their API. I guess they left the field for backwards compatibility when they migrated...

distutils: How to pass a user defined parameter to setup.py?

How can I pass a user-defined parameter both from the command line and setup.cfg configuration file to distutils' setup.py script?
I want to write a setup.py script, which accepts my package specific parameters. For example:
python setup.py install -foo myfoo

As Setuptools/Distuils are horribly documented, I had problems finding the answer to this myself. But eventually I stumbled across this example. Also, this similar question was helpful. Basically, a custom command with an option would look like:
from distutils.core import setup, Command
class InstallCommand(Command):
description = "Installs the foo."
user_options = [
('foo=', None, 'Specify the foo to bar.'),
]
def initialize_options(self):
self.foo = None
def finalize_options(self):
assert self.foo in (None, 'myFoo', 'myFoo2'), 'Invalid foo!'
def run(self):
install_all_the_things()
setup(
...,
cmdclass={
'install': InstallCommand,
}
)

Here is a very simple solution, all you have to do is filter out sys.argv and handle it yourself before you call to distutils setup(..).
Something like this:
if "--foo" in sys.argv:
do_foo_stuff()
sys.argv.remove("--foo")
...
setup(..)
The documentation on how to do this with distutils is terrible, eventually I came across this one: the hitchhikers guide to packaging, which uses sdist and its user_options.
I find the extending distutils reference not particularly helpful.
Although this looks like the "proper" way of doing it with distutils (at least the only one that I could find that is vaguely documented). I could not find anything on --with and --without switches mentioned in the other answer.
The problem with this distutils solution is that it is just way too involved for what I am looking for (which may also be the case for you).
Adding dozens of lines and subclassing sdist is just wrong for me.

Yes, it's 2015 and the documentation for adding commands and options in both setuptools and distutils is still largely missing.
After a few frustrating hours I figured out the following code for adding a custom option to the install command of setup.py:
from setuptools.command.install import install
class InstallCommand(install):
user_options = install.user_options + [
('custom_option=', None, 'Path to something')
]
def initialize_options(self):
install.initialize_options(self)
self.custom_option = None
def finalize_options(self):
#print('The custom option for install is ', self.custom_option)
install.finalize_options(self)
def run(self):
global my_custom_option
my_custom_option = self.custom_option
install.run(self) # OR: install.do_egg_install(self)
It's worth to mention that install.run() checks if it's called "natively" or had been patched:
if not self._called_from_setup(inspect.currentframe()):
orig.install.run(self)
else:
self.do_egg_install()
At this point you register your command with setup:
setup(
cmdclass={
'install': InstallCommand,
},
:

You can't really pass custom parameters to the script. However the following things are possible and could solve your problem:
optional features can be enabled using --with-featurename, standard features can be disabled using --without-featurename. [AFAIR this requires setuptools]
you can use environment variables, these however require to be set on windows whereas prefixing them works on linux/ OS X (FOO=bar python setup.py).
you can extend distutils with your own cmd_classes which can implement new features. They are also chainable, so you can use that to change variables in your script. (python setup.py foo install) will execute the foo command before it executes install.
Hope that helps somehow. Generally speaking I would suggest providing a bit more information what exactly your extra parameter should do, maybe there is a better solution available.

I successfully used a workaround to use a solution similar to totaam's suggestion. I ended up popping my extra arguments from the sys.argv list:
import sys
from distutils.core import setup
foo = 0
if '--foo' in sys.argv:
index = sys.argv.index('--foo')
sys.argv.pop(index) # Removes the '--foo'
foo = sys.argv.pop(index) # Returns the element after the '--foo'
# The foo is now ready to use for the setup
setup(...)
Some extra validation could be added to ensure the inputs are good, but this is how I did it

A quick and easy way similar to that given by totaam would be to use argparse to grab the -foo argument and leave the remaining arguments for the call to distutils.setup(). Using argparse for this would be better than iterating through sys.argv manually imho. For instance, add this at the beginning of your setup.py:
argparser = argparse.ArgumentParser(add_help=False)
argparser.add_argument('--foo', help='required foo argument', required=True)
args, unknown = argparser.parse_known_args()
sys.argv = [sys.argv[0]] + unknown
The add_help=False argument means that you can still get the regular setup.py help using -h (provided --foo is given).

Perhaps you are an unseasoned programmer like me that still struggled after reading all the answers above. Thus, you might find another example potentially helpful (and to address the comments in previous answers about entering the command line arguments):
class RunClientCommand(Command):
"""
A command class to runs the client GUI.
"""
description = "runs client gui"
# The format is (long option, short option, description).
user_options = [
('socket=', None, 'The socket of the server to connect (e.g. '127.0.0.1:8000')',
]
def initialize_options(self):
"""
Sets the default value for the server socket.
The method is responsible for setting default values for
all the options that the command supports.
Option dependencies should not be set here.
"""
self.socket = '127.0.0.1:8000'
def finalize_options(self):
"""
Overriding a required abstract method.
The method is responsible for setting and checking the
final values and option dependencies for all the options
just before the method run is executed.
In practice, this is where the values are assigned and verified.
"""
pass
def run(self):
"""
Semantically, runs 'python src/client/view.py SERVER_SOCKET' on the
command line.
"""
print(self.socket)
errno = subprocess.call([sys.executable, 'src/client/view.py ' + self.socket])
if errno != 0:
raise SystemExit("Unable to run client GUI!")
setup(
# Some other omitted details
cmdclass={
'runClient': RunClientCommand,
},
The above is tested and from some code I wrote. I have also included slightly more detailed docstrings to make things easier to understand.
As for the command line: python setup.py runClient --socket=127.0.0.1:7777. A quick double check using print statements shows that indeed the correct argument is picked up by the run method.
Other resources I found useful (more and more examples):
Custom distutils commands
https://seasonofcode.com/posts/how-to-add-custom-build-steps-and-commands-to-setuppy.html

To be fully compatible with both python setup.py install and pip install . you need to use environment variables because pip option --install-option= is bugged:
pip --install-option leaks across lines
Determine what should be done about --(install|global)-option with Wheels
pip not naming abi3 wheels correctly
This is a full example not using the --install-option:
import os
environment_variable_name = 'MY_ENVIRONMENT_VARIABLE'
environment_variable_value = os.environ.get( environment_variable_name, None )
if environment_variable_value is not None:
sys.stderr.write( "Using '%s=%s' environment variable!\n" % (
environment_variable_name, environment_variable_value ) )
setup(
name = 'packagename',
version = '1.0.0',
...
)
Then, you can run it like this on Linux:
MY_ENVIRONMENT_VARIABLE=1 pip install .
MY_ENVIRONMENT_VARIABLE=1 pip install -e .
MY_ENVIRONMENT_VARIABLE=1 python setup.py install
MY_ENVIRONMENT_VARIABLE=1 python setup.py develop
But, if you are on Windows, run it like this:
set "MY_ENVIRONMENT_VARIABLE=1" && pip install .
set "MY_ENVIRONMENT_VARIABLE=1" && pip install -e .
set "MY_ENVIRONMENT_VARIABLE=1" && python setup.py install
set "MY_ENVIRONMENT_VARIABLE=1" && python setup.py develop
References:
How to obtain arguments passed to setup.py from pip with '--install-option'?
Passing command line arguments to pip install
Passing the library path as a command line argument to setup.py