Develop Reference

youtube-dl setting options in python

I am trying to save livestreams using youtube-dl API in python with the following code. Since it's a continuous live stream there is no end to the video, so I am using hls-use-mpegts as a way to periodically read the video for processing, that flag makes .mp4.part files playable.
Although the hls-use-mpegts option works well with the command-line thus:
youtube-dl -f worst <some URL> --retries infinite --continue --hls-use-mpegts
it doesn't seem to work with this code. I don't see any errors but don't see the file being saved in mpegts format. Do I have the options setting correct?
ydl_opts = {
'format': 'worst',
'retries': 99,
'continue': True,
'hls-use-mpegts': True
}
with youtube_dl.YoutubeDL(ydl_opts) as ydl:
ydl.download([url])

It's because (sorry for saying that) the docs is somewhat good&shitt* at the same time.
I found every switches/cli-options that you were to use within Python you have to replace - (dash) to _ (sub dash).
Solution
In your case, hls_use_mpegts is the solution.
Why?
Read/explore about that here: https://github.com/ytdl-org/youtube-dl/blob/5208ae92fc3e2916cdccae45c6b9a516be3d5796/youtube_dl/downloader/common.py#L50
and
here: https://github.com/ytdl-org/youtube-dl/blob/5208ae92fc3e2916cdccae45c6b9a516be3d5796/youtube_dl/__init__.py#L428
or just browse as I do usually for those inconveniences: https://github.com/ytdl-org/youtube-dl/search?q=hls_use_mpegts%3A (fortunately GitHub does a really good job at this, and u don't have to have the src code downloaded to be searched)
Otherwise it's fun to use yt-dl, thanks for them!

Jupyter nbconvert LaTex Export Theme

I am using Jupyter Notebook nbconvert (Save as menu) to export as pdf via Latex. However, the pdf file is not in a good shape. For example, some wide tables are shown well. I would prefer to have a box for tables to be resized to the width of the page. Is there any style, template that I can use to have nice reports and how I may ask nbconverter to use that style?
Here is the Latex output:
I would like something like this:

Looks like Pandas gained a ._repr_latex_() method in version 0.23. You'll need to set pd.options.display.latex.repr=True to activate it.
Without latex repr:
With latex repr:
Check out the options to get the formatting close to what you want. In order to match your desired output exactly, you'll need to use a custom latex template.
Edited to provide more information on templates:
Start here for general information about templates. You can create a .tplx file in the same path as your notebook and specify it as the template when running nbconvert from the command line: !jupyter nbconvert --to python 'example.ipynb' --stdout --template=my_custom_template.tplx. Alternatively, you can specify a default template to use when exporting as Latex via the menu by modifying the jupyter_notebook_config.py file in your ~.jupyter directory. If this file doesn't exist already, you can generate it by running the command jupyter notebook --generate-config from the command line. I have my template sitting in the ~/.jupyter directory as well, so I added the following to my jupyter_notebook_config.py:
# Insert this at the top of the file to allow you to reference
# a template in the ~.jupyter directory
import os.path
import sys
sys.path.insert(0, os.path.expanduser("~") + '/.jupyter')
# Insert this at the bottom of the file:
c.LatexExporter.template_file = 'my_template' # no .tplx extension here
c.LatexExporter.template_path = ['.', os.path.expanduser("~") + '/.jupyter'] # nbconvert will look in ~/.jupyter
To understand a bit about how the templates work, start by taking a look at null.tplx. The line ((*- for cell in nb.cells -*)) loops over all the cells in the notebook. The if statements that follow check the type of each cell and call the appropriate block.
The other templates extend null.tplx. Each template defines (or redefines) some of the blocks. The hierarchy is null->display_priority->document_contents->base->style_*->article.
Your custom template should probably extend article.tplx and add some Latex commands to the header that sets up the tables the way you want. Take a look at this blog post for an example of setting up a custom template.

Any setting that change the table size to fit it in the width of the page?
Latex code is something like this: \resizebox*{\textwidth}{!}{%

How can I handle reading a .json file in it that has comments with python?

Firstly, I understand that comments aren't valid json. That said, for some reason this .json file I have to process has comments at the start of lines and at the end of lines.
How can i handle this in python and basically load the .json file but ignore the comments so that I can process it? I am currently doing the following:
with open('/home/sam/Lean/Launcher/bin/Debug/config.json', 'r') as f:
config_data=json.load(f)
But this crashes at the json.load(f) command because the file f has comments in it.
I thought this would be a common problem but I can't find much online RE how to handle it in python. Someone suggested commentjson but that makes my script crash saying
ImportError: cannot import name 'dump'
When I import commentjson
Thoughts?
Edit:
Here is a snippet of the json file i must process.
{
// this configuration file works by first loading all top-level
// configuration items and then will load the specified environment
// on top, this provides a layering affect. environment names can be
// anything, and just require definition in this file. There's
// two predefined environments, 'backtesting' and 'live', feel free
// to add more!
"environment": "backtesting",// "live-paper", "backtesting", "live-interactive", "live-interactive-iqfeed"
// algorithm class selector
"algorithm-type-name": "BasicTemplateAlgorithm",
// Algorithm language selector - options CSharp, FSharp, VisualBasic, Python, Java
"algorithm-language": "CSharp"
}

Switch into json5. The JSON 5 is a very small superset of JSON that supports comments and few other features you could just ignore.
import json5 as json
# and the rest is the same
It is beta, and it is slower, but if you just need to read some short configuration once when starting the program, this probably can be considered as an option. It is better to switch into another standard than not to follow any.

kind of a hack (because if there are // within the json data then it will fail) but simple enough for most cases:
import json,re
s = """{
// this configuration file works by first loading all top-level
// configuration items and then will load the specified environment
// on top, this provides a layering affect. environment names can be
// anything, and just require definition in this file. There's
// two predefined environments, 'backtesting' and 'live', feel free
// to add more!
"environment": "backtesting",// "live-paper", "backtesting", "live-interactive", "live-interactive-iqfeed"
// algorithm class selector
"algorithm-type-name": "BasicTemplateAlgorithm",
// Algorithm language selector - options CSharp, FSharp, VisualBasic, Python, Java
"algorithm-language": "CSharp"
}
"""
result = json.loads(re.sub("//.*","",s,flags=re.MULTILINE))
print(result)
gives:
{'environment': 'backtesting', 'algorithm-type-name': 'BasicTemplateAlgorithm', 'algorithm-language': 'CSharp'}
apply regular expression to all the lines, removing double slashes and all that follows.
Maybe a state machine parsing the line would be better to make sure the // aren't in quotes, but that's slightly more complex (but doable)

I haven't used it personally but you can have a look on JSONComment python package which supports parsing a json file with comment. Use it in place of JsonParser
parser = JsonComment(json)
parsed_object = parser.loads(jsonString)

You can take out the comments with the following:
data=re.sub("//.*?\n","",data)
data=re.sub("/\\*.*?\\*/","",data)
This should remove all comments from the data. It could cause problems if there are // or /* inside your strings

How do I save a Beaker notebook as straight python/r/...?

I just discovered Beaker Notebook. I love the concept, and am desperately keen to use it for work. To do so, I need to be sure I can share my code in other formats.
Question
Say I write pure Python in a Beaker notebook:
Can I save it as a .py file as I can in iPython Notebook/Jupyter?
Could I do the same if I wrote a pure R Beaker notebook?
If I wrote a mixed (polyglot) notebook with Python and R, can I save this to e.g. Python, with R code present but commented out?
Lets say none of the above are possible. Looking at the Beaker Notebook file as a text file, it seems to be saved in JSON. I can even find the cells that correspond to e.g. Python, R. It doesn't look like it would be too challenging to write a python script that does 1-3 above. Am I missing something?
Thanks!
PS - there's no Beaker notebook tag!? bad sign...

It's really not that hard to replicate the basics of the export:
#' Save a beaker notebook cell type to a file
#'
#' #param notebook path to the notebook file
#' #param output path to the output file (NOTE: this file will be overwritten)
#' #param cell_type which cells to export
save_bkr <- function(notebook="notebook.bkr",
output="saved.py",
cell_type="IPython") {
nb <- jsonlite::fromJSON(notebook)
tmp <- subset(nb$cells, evaluator == cell_type)$input
if (length(tmp) != 0) {
unlink(output)
purrr::walk(tidyr::unnest(tmp, body), cat, file=output, append=TRUE, sep="\n")
} else {
message("No cells found matching cell type")
}
}
I have no idea what Jupyter does with the "magic" stuff (gosh how can notebook folks take that seriously with a name like "magic").
This can be enhanced greatly, but it gets you the basics of what you asked.

MkDocs and MathJax

I'm new to MkDocs and am writing some technical documentation that requires latex. I've successfully built a small website with one of the MkDocs themes, however it won't properly display the latex equations. I followed the instructions at:
http://www.vlfeat.org/matconvnet/developers/
as well as the instructions following the python-markdown-mathjax link from that page. I have also tinkered with adding appropriate lines to my mkdocs.yaml file, similar to:
https://github.com/EdyJ/vehicle-physics-docs/blob/master/mkdocs.yml
However, issuing the command 'mkdocs build' still results in a site that doesn't render the equations. I've also tried adding a -x mathjax flag with the mkdocs build command.
I've scoured the web and have been tinkering for quite a bit of time now. Can anyone shed light on what I need to do to get these two playing together?

This is actually easier than I expected. First I installed the Python-Markdown-Math Extension:
pip install https://github.com/mitya57/python-markdown-math/archive/master.zip
Then I created a new MkDocs project:
mkdocs new test_math
Next I edited the test_math/docs/index.md file to be as follows (sample borrowed from the MathJax documentation):
# MathJax Test Page
When \(a \ne 0\), there are two solutions to \(ax^2 + bx + c = 0\) and they are
$$x = {-b \pm \sqrt{b^2-4ac} \over 2a}.$$
Finally, I edited the test_math/config.yaml file to be as follows:
site_name: Test Math
extra_javascript:
- https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML
markdown_extensions:
- mdx_math
I was unsure if this would work, but I ran the test server to see:
mkdocs serve
I then opened my browser and loaded http://127.0.0.1:8000/. The page displayed with the sample equations properly formatted:
Then I remembered that the OP asked for this to work with ReadTheDocs, so I added the following line to the config:
theme: readthedocs
My browser reloaded and the following (properly formatted equations) displayed:
I should note that I'm getting some weird error about fontawesome not loading. With the MkdDocs' theme, the equations disappear after a minute (when the error appears in the browser's console). However, in the ReadTheDocs theme, the equations display properly, even with the error. Either way, I believe this error is related to some other issue on my local machine.
Finally, the Bounty is...
Looking for an answer drawing from credible and/or official sources
I don't normally advertise this, but since you asked, I am the lead developer of Python-Markdown, I work regularly with mitya57 (the creator of Python-Markdown-Math Extension) as he is one of two other developers with commit access to Python-Markdown, and I am a contributor to MkDocs (one of those contributions being support for Python-Markdown Extensions).

You will need to install the extension as indicated in the github README. It sounds as if you have already done that.
Then you need to tell Mkdocs that you are using this Python Markdown extension. You would do this by having a line such as this in your mkdocs.yaml:
markdown_extensions: [mathjax]
The YAML configuration documentation can be found at:
http://www.mkdocs.org/user-guide/configuration/#formatting-options

I am no expert on any of this but the below is what worked for me. One of my needs was to have the $...$ notation work for inline Latex, instead of \(..\) because the $ notation works directly in Jupyter notebooks and you can see what your text will look like without first running mkdocs.
For both $..$ and $$..$$ styles to work, first install pip install --upgrade python-markdown-math. Then do the following:
Put a text file called mathjaxhelper.js in the /docs folder, and it should contain only the following:
MathJax.Hub.Config({
config: ["MMLorHTML.js"],
jax: ["input/TeX", "output/HTML-CSS", "output/NativeMML"],
extensions: ["MathMenu.js", "MathZoom.js"]
});
The project.yml file should contain the following. (replace project.yml with your actual yml file)
markdown_extensions:
- extra
- tables
- mdx_math:
enable_dollar_delimiter: True
- fenced_code
theme: readthedocs
extra_javascript:
- https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.4/MathJax.js?config=TeX-AMS-MML_HTMLorMML
I will admit I am no expert and really only cared about what worked. It took me several hours to figure out what combination of things in the .yml and the mathjaxhelper.js file will get both $ and $$ to work. Hope this helps someone else.

It looks like be that this extension is not required:
docs/mathjaxhelper.js
MathJax.Hub.Config({
"tex2jax": { inlineMath: [ [ '$', '$' ] ] }
});
MathJax.Hub.Config({
config: ["MMLorHTML.js"],
jax: ["input/TeX", "output/HTML-CSS", "output/NativeMML"],
extensions: ["MathMenu.js", "MathZoom.js"]
});
mkdocs.yml
markdown_extensions:
- extra
- tables
- fenced_code
extra_javascript:
- https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS_HTML
- mathjaxhelper.js
Seems to do the trick.

Here is a way to get LaTeX in MkDocs with the extension arithmatex.
I'm answering again because I want to document a solution that doesn't link to third party sites like Cloudflare and Google.
I'm going the way with the theme named mkdocs-material but you can replace theme: ...school with theme: readthedocs.
First install MkDocs and create a project:
https://www.mkdocs.org/#installation.
Then install mkdocs-material and MathJax as follows.
Say the project is called my-project.
Use downloads and not pip because you want to change the CDN to Cloudflare and the link to google.
cd my-project
git clone https://github.com/squidfunk/mkdocs-material.git
git clone https://github.com/mathjax/MathJax.git MathJax
Edit mkdocs.yml like described under "Usage", for example:
site_name: My Project
pages:
- Home: index.md
theme:
name: null
custom_dir: 'mkdocs-material/material'
palette:
primary: amber
accent: pink
language: de
feature:
tabs: true
font: false
logo:
icon: school
markdown_extensions:
- pymdownx.arithmatex
extra_javascript:
- ../MathJax/MathJax.js?config=TeX-AMS-MML_HTMLorMML
You can insert some TeX in index.md, build the site (mkdocs build) and open the index.html to check that MathJax works.
Because it doesn't work with mkdocs serve I uploaded the MathJax folder and linking to it instead of the link to the local path.
extra_javascript:
- https://mysite/MathJax/MathJax.js?config=TeX-AMS-MML_HTMLorMML
To activate TeX inline with $...$ you have to create an extra JS-file
cd docs
mkdir assets
touch extra.js
Insert the following text into extra.js:
window.MathJax = {
tex2jax: {
inlineMath: [ ["$","$"], ["\\(","\\)"] ],
displayMath: [ ["\\[","\\]"] ]
},
TeX: {
TagSide: "right",
TagIndent: ".8em",
MultLineWidth: "85%",
equationNumbers: {
autoNumber: "AMS",
},
unicode: {
fonts: "STIXGeneral,'Arial Unicode MS'"
}
},
displayAlign: "center",
showProcessingMessages: false,
messageStyle: "none"
};
and link to it in mkdocs.yml. Finaly the extra_javascript section can like this:
extra_javascript:
- https://mysite/MathJax/MathJax.js?config=TeX-AMS-MML_HTMLorMML
- assets/extra.js
Because we don't want to use CDN you can edit mkdocs-material/material/base.html and delete/uncomment the line with <link href="https://fonts.gstatic.com" rel="preconnect" crossorigin>.
Font-awesome and material-icons are already on board within mkdocs-material.