Develop Reference

flake8's "# noqa" interferes with markdown using mkdocstrings

I am using mkdocstrings in order automatically generate an API documentation from my Python functions. At the same time I am using flake8 to keep my code in good shape.
If you want to ignore some flake8 warnings on an in-line basis, you could insert "# noqa" whereby the following lines of code will be ignored by flake8.
That's nice, however, "# noqa" will be interpreted by mkdocstrings as a markdown header.
Now, I am wondering how to resolve that conflict between flake8 and mkdocstrings?

put the noqa comment on the end of the docstring -- it will apply to any line within the docstring without changing the string's contents (note: you need a sufficiently new flake8, this change is relatively recent (probably >=4.x))
def f():
"""some docstring here
something which causes a warning
""" # noqa: ABC123
disclaimer: I am the current flake8 maintainer

Issues rendering MathJax using Sphinx due to HTML-CSS renderer

I just upgraded to the following:
Sphinx==1.8.5
nbconvert==5.4.1
pandoc==1.0.2
I have always been able to render math in docs via a jupyter notebook using Sphinx (the way it is done for seaborn). However, after upgrading, the math no longer renders! If I right click where the equation should be, I can change the math renderer to either "svg" or "CommonHTML" and everything is fine - it just doesn't render with the default "HTML-CSS" math renderer for whatever reason. My question is: how can I change the default renderer in my config file?
What I tried:
conf.py
...
extensions = [
...
'sphinx.ext.mathjax'
...
]
...
mathjax_config = {
'jax': ['input/TeX', 'output/CommonHTML']
}
However, this didn't change the default renderer like I thought it should. Does anyone know how to change the default renderer for MathJax within Sphinx?
Here is a gif:
Also, it would appear that MathJax is being loaded from CDN with a preferred config:

I am not sure how MathJax worked for you before, but based on the Sphinx-documentation MathJax is not included in Sphinx by default. You have to specify the mathjax_path in your conf.py either using a local version or a server hosted one from cdnjs for example.

While I wasn't able to get the mathjax_config approach to work, I was able to set the mathjax_path to a specific output processor, still through the CDN, which worked using CommonHTML.
So, an answer that works is to edit the conf.py file to include a line with:
...
mathjax_path = "https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-MML-AM_CHTML"
...

Add usage help of command line tool to README.rst

I wrote a little command line tool, and want to add the "--help" usage message to the docs.
Since I am lazy, I would like to make the update procedure as simple as possible. Here is what I want to the update workflow to look like:
Update code which results in an updates usage message.
Run a script which updates the docs: The new usage message should be visible in the docs.
In other word: I don't want to copy+paste the usage message.
Step1 is comes from my own brain. But want to reuse existing tools for Step2.
Up to now the docs are just a simple README.rst file.
I would like to stick with a simple solution, where the docs can be visible directly via github. Up to now, I don't need the more complicated solution (like readthedocs).
How can I avoid copy+pasting the --help usage message?
Here is the tool I am working on: https://github.com/guettli/reprec

As suggested in the comments, you could use a git pre-commit hook to generate the README.rst file on commit. You could use an existing tool such as cog, or you could just do something very simple with bash.
For example, create a RST "template" file:
README.rst.tmpl
Test Git pre-commit hook project
--------------------------------
>>> INSERTION POINT FOR HELP OUTPUT <<<
.git/hooks/pre-commit
# Sensible to set -e to ensure we exit if anything fails
set -e
# Get the output from your tool.
# Paths are relative to the root of the repo
output=$(tools/my-cmd-line-tool --help)
cat README.rst.tmpl |
while read line
do
if [[ $line == ">>> INSERTION POINT FOR HELP OUTPUT <<<" ]]
then
echo "$output"
else
echo "$line"
fi
done > README.rst
git add README.rst
This gets run before you are prompted for a commit message, if you didn't pass one on the command line. So when the commit takes place if there were any changes to either README.rst.tmpl or the output from your tool, README.rst will be updated with it.
Edit
I believe this should work on Windows too, or something very similar, since git comes with a bash implementation on Windows, but I haven't tested it.

Consider using cog. It's meant for exactly this job.
Here's something that might just work. (untested) And... There's a lot of scope for improvement.
reprec
======
The tool reprec replaces strings in text files:
.. [[[cog
.. import cog
..
.. def indent(text, width=4):
.. return "\n".join((" "*width + line) for line in text.splitlines())
..
.. text = subprocess.check_output("reprec --help", shell=True)
.. cog.out("""
..
.. ::
..
.. ==> reprec --help""",
.. dedent=True
.. )
.. cog.out(indent(text))
.. ]]]
::
===> reprec --help
<all-help-text>
.. [[[end]]]

For getting the usage text at Step 2, you can use the subprocess
usage_text = subprocess.check_output("reprec --help", shell=True)

I would actually approach in a quite different manner, from another side. I think the workflow you described may be greatly simplified if you switch to using argparse instead of getopt you use now. With this you will have:
I personally think, simpler code in your argument parsing function, and probably more safe, because argparse may verify a lot of conditions on given arguments, as long as you declare them (like data types, number of arguments, etc.)
and you can use argparse features to document the arguments directly in the code, right where you declare them (e.g.: help, usage, epilog and others); this effectively means that you could completely delete your own usage function, because argparse will handle this task for you (just run with --help to see the result).
To sum up, basically, arguments, their contracts and help documentation become mostly declarative, and managed altogether in one place only.
OK, OK, I know, the question originally stands how to update the README. I understand that your intention is to take the laziest approach. So, I think, it is lazy enough to:
maintain all your arguments and their documentation once in single place as above
then run something like myprograom --help > README.rst
commit ;)
OK, you will probably need something little bit more complex than just > README.rst. There we can go creative as we want, so the fun starts here. For example:
having README.template.rst (where you actually maintain the README content) and with ## Usage header somewhere in it:
$ myprogram --help > USAGE.rst
$ sed -e '/## Usage/r USAGE.rst' -e '$G' README.template.rst > README.rst
And you get everything working from same source code!
I think it will still need some polishing up, in order to generate valid rst document, but I hope it shows the idea in general.
Gist: Include generated help into README

Ignore the rest of the python file

My python scripts often contain "executable code" (functions, classes, &c) in the first part of the file and "test code" (interactive experiments) at the end.
I want python, py_compile, pylint &c to completely ignore the experimental stuff at the end.
I am looking for something like #if 0 for cpp.
How can this be done?
Here are some ideas and the reasons they are bad:
sys.exit(0): works for python but not py_compile and pylint
put all experimental code under def test():: I can no longer copy/paste the code into a python REPL because it has non-trivial indent
put all experimental code between lines with """: emacs no longer indents and fontifies the code properly
comment and uncomment the code all the time: I am too lazy (yes, this is a single key press, but I have to remember to do that!)
put the test code into a separate file: I want to keep the related stuff together
PS. My IDE is Emacs and my python interpreter is pyspark.

Use ipython rather than python for your REPL It has better code completion and introspection and when you paste indented code it can automatically "de-indent" the pasted code.
Thus you can put your experimental code in a test function and then paste in parts without worrying and having to de-indent your code.
If you are pasting large blocks that can be considered individual blocks then you will need to use the %paste or %cpaste magics.
eg.
for i in range(3):
i *= 2
# with the following the blank line this is a complete block
print(i)
With a normal paste:
In [1]: for i in range(3):
...: i *= 2
...:
In [2]: print(i)
4
Using %paste
In [3]: %paste
for i in range(10):
i *= 2
print(i)
## -- End pasted text --
0
2
4
In [4]:
PySpark and IPython
It is also possible to launch PySpark in IPython, the enhanced Python interpreter. PySpark works with IPython 1.0.0 and later. To use IPython, set the IPYTHON variable to 1 when running bin/pyspark:1
$ IPYTHON=1 ./bin/pyspark

Unfortunately, there is no widely (or any) standard describing what you are talking about, so getting a bunch of python specific things to work like this will be difficult.
However, you could wrap these commands in such a way that they only read until a signifier. For example (assuming you are on a unix system):
cat $file | sed '/exit(0)/q' |sed '/exit(0)/d'
The command will read until 'exit(0)' is found. You could pipe this into your checkers, or create a temp file that your checkers read. You could create wrapper executable files on your path that may work with your editors.
Windows may be able to use a similar technique.
I might advise a different approach. Separate files might be best. You might explore iPython notebooks as a possible solution, but I'm not sure exactly what your use case is.

Follow something like option 2.
I usually put experimental code in a main method.
def main ():
*experimental code goes here *
Then if you want to execute the experimental code just call the main.
main()

With python-mode.el mark arbitrary chunks as section - for example via py-sectionize-region.
Than call py-execute-section.
Updated after comment:
python-mode.el is delivered by melpa.
M-x list-packages RET
Look for python-mode - the built-in python.el provides 'python, while python-mode.el provides 'python-mode.
Developement just moved hereto: https://gitlab.com/python-mode-devs/python-mode

I think the standard ('Pythonic') way to deal with this is to do it like so:
class MyClass(object):
...
def my_function():
...
if __name__ == '__main__':
# testing code here
Edit after your comment
I don't think what you want is possible using a plain Python interpreter. You could have a look at the IEP Python editor (website, bitbucket): it supports something like Matlab's cell mode, where a cell can be defined with a double comment character (##):
## main code
class MyClass(object):
...
def my_function():
...
## testing code
do_some_testing_please()
All code from a ##-beginning line until either the next such line or end-of-file constitutes a single cell.
Whenever the cursor is within a particular cell and you strike some hotkey (default Ctrl+Enter), the code within that cell is executed in the currently running interpreter. An additional feature of IEP is that selected code can be executed with F9; a pretty standard feature but the nice thing here is that IEP will smartly deal with whitespace, so just selecting and pasting stuff from inside a method will automatically work.

I suggest you use a proper version control system to keep the "real" and the "experimental" parts separated.
For example, using Git, you could only include the real code without the experimental parts in your commits (using add -p), and then temporarily stash the experimental parts for running your various tools.
You could also keep the experimental parts in their own branch which you then rebase on top of the non-experimental parts when you need them.

Another possibility is to put tests as doctests into the docstrings of your code, which admittedly is only practical for simpler cases.
This way, they are only treated as executable code by the doctest module, but as comments otherwise.

XCode Formatting issue for python files

I have used xcode to create several python scripts. It appears in the editor fine, however, when I attempt to look at the code through the terminal, I'm seeing that new lines are being encoded as "^M". This is problematic, since I am collaborating through github, and the diff features do not work when this is being done.
E.g.:
Source:
#############
#
# test.py
#
# by Author
#
#############
if __name__ == "__main__":
print "This is a test"
When I save this through another editor (PyCharmer) and more it via the console, I get the output as expected. When I create a new file via xcode, past the same text, and save, I get the following:
#############^M#^M# test.py^M#^M# by Author^M#^M#############^Mif __name__ == "__main__":^M print "This is a test"
Out of curiosity, I tried creating a test .cc file, and the same formatting issue did not arise, so if you want bonus points, explaining the inconsistency would be interesting as well.

In the latest Xcode 6, if I create a new external-build-system project, set the build tool to /usr/bin/python or /usr/local/bin/python3, create a new file named test.py in that project, Xcode recognizes its type (which you can see in the File Inspector panel of the assistant editor) as "Default — Python script", and its text settings (which you can also see in the File Inspector) as:
Text Encoding: Unicode (UTF-8)
Line Endings: Default — OS X / Unix (LF)
Indent Using: Spaces
Widths: Tab: 4 Indent: 4
Wrap lines: checked
And I get Python syntax coloring, tab completions, etc.
If that's not right—in particular, if you see Line Endings as "Classic Mac OS (CR)"—you can change it for the current file right there in the panel.
That fixes the current file. It may not fix the next file you create, but try it and see.
If not: In Xcode 4 and 5, you could easily change the default settings for each language, but that no longer seems to be exposed in Xcode 6. However, you may want to try going to the "Text Editing" pane of the "Preferences" dialog, and making sure "Default line endings" is set to "OS X / Unix (LF)", and maybe that "Convert existing files on save" is checked. This will help if you've got your default settings to Classic Mac, but C/ObjC/C++ overriding that with Unix. If, on the other hand, you somehow have a leftover override for Python from an earlier version of Xcode, I'm not sure how you can undo it short of wiping all of your Xcode settings and starting clean.
From a quick search, this answer has a very detailed version of some of the steps involved in configuring Python projects to handle things like the Run and Debug commands and so on, which may also be tangentially helpful.