I've made a custom nbconvert template and want it to be accessible from any folder where I launch nbconvert utility. Where should I put my template?
I couldn't find anything in official docs. I have already tried usual places for jupyter configs, like /usr/share/jupyter, ~/.local/share/jupyter, ~/.jupyter, to no avail.
The only place I've found so far is the folder where python package lives:
$ pip show nbconvert | grep Location | cut -d" " -f2
/usr/lib/python3.6/site-packages
If I create nbconvert/templates/html directory there and put my template in it, nbconvert --to html --template <my_template_name> ... works fine. But this is an ugly hack which I'll need to re-do every time I update nbconvert.
Seems that I can provide nbconvert with environment variable, but I would prefer to avoid this option.
You need to tell nbconvert to look for your template by creating an jupyter_nbconvert_config.py file and storing it in ~/.jupyter.
I use this for LaTeX--here's what my file looks like:
import os
c = get_config()
c.LatexExporter.template_path = ['.', os.path.expanduser('~/.jupyter/templates')]
c.LatexExporter.template_file = 'custom_latex.tplx'
Assuming you template extends an existing one, you need to include '.' when setting template_path so it knows where to look for the standard templates.
From the docs.
The recommended place to save custom templates, so that they are globally accessible to nbconvert, is your jupyter data directories:
share/jupyter
nbconvert
templates
html
latex
Alternately
from jupyter_core.paths import jupyter_path
print(jupyter_path('nbconvert','templates'))
I encountered this problem when installing nbconvert to a custom location using:
pip install --target=/foooooo/baaaaar nbconvert
You just need to set a JUPYTER_PATH environment variable.
JUPYTER_PATH=/foooooo/baaaaar/share/jupyter
As an alternative to editing jupyter_nbconvert_config.py you can also edit jupyter_nbconvert_config.json. First assert that ~/.jupyter is in the config path with jupyter --path. Then insert in jupyter_nbconvert_config.json a template directory. I added a subfolder custome_templates to mine:
{
"Exporter": {
"template_path": [
".",
"/home/moutsopoulosg/miniconda/envs/myenv/lib/python2.7/site-packages/jupyter_contrib_nbextensions/templates",
"/home/moutsopoulosg/.jupyter/custom_templates"
],
...
},
"version": 1
}
Then nbconvert --template mytemplate Untitiled.ipynb picks up my template.
Related
I know there have been a few questions about this but I have not found anything robust enough.
Currently I am using, from terminal, a command that creates .py, then moves them to another folder:
jupyter nbconvert --to script '/folder/notebooks/notebook.ipynb' && \
mv ./folder/notebooks/*.py ./folder/python_scripts && \
The workflow then is to code in a notebook, check with git status what changed since last commit, create a potentially huge number of nbconvert commands, then move them all.
I would like to use something like !jupyter nbconvert --to scriptfound in this answer, but without the cell that crates the python file appearing in the .py itself.
Because if that line appears, my code won't ever work right.
So, is there a proper way of dealing with this problem? One that can be automated, and not manually copying files names, creating the command, executing and then starting again.
You can add the following code in the last cell in your notebook file.
!jupyter nbconvert --to script mycode.ipynb
with open('mycode.py', 'r') as f:
lines = f.readlines()
with open('mycode.py', 'w') as f:
for line in lines:
if 'nbconvert --to script' in line:
break
else:
f.write(line)
It will generate the .py file and then remove this very code from it. You will end up with a clean script that will not call !jupyter nbconvert anymore.
Another way would be to use Jupytext as extension for your jupyter installation (can be easily pip installed).
Jupytext Description (see github page)
Have you always wished Jupyter notebooks were plain text documents?
Wished you could edit them in your favorite IDE? And get clear and
meaningful diffs when doing version control? Then... Jupytext may well
be the tool you're looking for!
It will keep paired notebooks in sync with .py files. You then just need to move your .py files or gitignore the notebooks for example as possible workflows.
Go to File > Save and Export Notebook as... > Executable Scripts
This is the closest I have found to what I had in mind, but I have yet to try and implement it:
# A post-save hook to make a script equivalent whenever the notebook is saved (replacing the --script option in older versions of the notebook):
import io
import os
from notebook.utils import to_api_path
_script_exporter = None
def script_post_save(model, os_path, contents_manager, **kwargs):
"""convert notebooks to Python script after save with nbconvert
replaces `jupyter notebook --script`
"""
from nbconvert.exporters.script import ScriptExporter
if model['type'] != 'notebook':
return
global _script_exporter
if _script_exporter is None:
_script_exporter = ScriptExporter(parent=contents_manager)
log = contents_manager.log
base, ext = os.path.splitext(os_path)
script, resources = _script_exporter.from_filename(os_path)
script_fname = base + resources.get('output_extension', '.txt')
log.info("Saving script /%s", to_api_path(script_fname, contents_manager.root_dir))
with io.open(script_fname, 'w', encoding='utf-8') as f:
f.write(script)
c.FileContentsManager.post_save_hook = script_post_save
Additionally, this looks like it has worked to some user on github, so I put it here for reference:
import os
from subprocess import check_call
def post_save(model, os_path, contents_manager):
"""post-save hook for converting notebooks to .py scripts"""
if model['type'] != 'notebook':
return # only do this for notebooks
d, fname = os.path.split(os_path)
check_call(['ipython', 'nbconvert', '--to', 'script', fname], cwd=d)
im trying to convert four python files that are related (belongs to the same project) into a jupyter notebook(ipynb) one file , is there any specific way to do that ?
This is my project folder tree:
C:/
build_dataset.py
train_model.py
folder1
---cancernet.py
---config.py
dataset_folder
You can use py2nb tool for it:
https://github.com/williamjameshandley/py2nb
Just call it from the shell:
py2nb waka.py
and you will get the .ipynb file.
PS: There are several similar tools. p2j also can help you. Usage is absolutely equal to py2nb. Or you can use the powerful jupytext with its command line conversions between formats:
jupytext --to notebook notebook.py # overwrite notebook.ipynb (remove outputs)
jupytext --to notebook --update notebook.py # update notebook.ipynb (preserve outputs)
jupytext --to ipynb notebook1.md notebook2.py # overwrite notebook1.ipynb and notebook2.ipynb
I'm trying to understand how nix works. For that purposed I tried to create a simple environment to run jupyter notebooks.
When I run the command:
nix-shell -p "\
with import <nixpkgs> {};\
python35.withPackages (ps: [\
ps.numpy\
ps.toolz\
ps.jupyter\
])\
"
I get what I expected -- a shell in an environment with python and the all packages listed installed, and the all expected commands accessible in the path:
[nix-shell:~/dev/hurricanes]$ which python
/nix/store/5scsbf8z3jnz8ardch86mhr8xcyc8jr2-python3-3.5.3-env/bin/python
[nix-shell:~/dev/hurricanes]$ which jupyter
/nix/store/5scsbf8z3jnz8ardch86mhr8xcyc8jr2-python3-3.5.3-env/bin/jupyter
[nix-shell:~/dev/hurricanes]$ jupyter notebook
[I 22:12:26.191 NotebookApp] Serving notebooks from local directory: /home/calsaverini/dev/hurricanes
[I 22:12:26.191 NotebookApp] 0 active kernels
[I 22:12:26.191 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/?token=7424791f6788af34f4c2616490b84f0d18353a4d4e60b2b5
So, I created a new folder with a single default.nix file with the following contents:
with import <nixpkgs> {};
python35.withPackages (ps: [
ps.numpy
ps.toolz
ps.jupyter
])
When I run nix-shell in this folder though, it seems like everything is installed but the PATHs are not set:
[nix-shell:~/dev/hurricanes]$ which python
/usr/bin/python
[nix-shell:~/dev/hurricanes]$ which jupyter
[nix-shell:~/dev/hurricanes]$ jupyter
The program 'jupyter' is currently not installed. You can install it by typing:
sudo apt install jupyter-core
By what I read here I was expecting the two situations to be equivalent. What did I do wrong?
Your default.nix file is supposed to hold the information to build a derivation when calling it with nix-build. When calling it with nix-shell, it just sets the shell in a way that the derivation is buildable. In particular, it sets the PATH variable to contain everything that is listed in the buildInput attribute:
with import <nixpkgs> {};
stdenv.mkDerivation {
name = "my-env";
# src = ./.;
buildInputs =
python35.withPackages (ps: [
ps.numpy
ps.toolz
ps.jupyter
]);
}
Here, I've commented out the src attribute which is required if you want to run nix-build but isn't necessary when your are just running nix-shell.
In your last sentence, I suppose you are referring more precisely to this:
https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#load-environment-from-nix-expression
I don't understand this advice: to me it just looks plain false.
Is there any way to get pip to print the config it will attempt to use? For debugging purposes it would be very nice to know that:
config.ini files are in the correct place and pip is finding them.
The precedence of the config settings is treated in the way one would expect from the docs
For 10.0.x and higher
There is new pip config command, to list current configuration values
pip config list
(As pointed by #wmaddox in comments) To get the list of where pip looks for config files
pip config list -v
Pre 10.0.x
You can start python console and do. (If you have virtaulenv don't forget to activate it first)
from pip import create_main_parser
parser = create_main_parser()
# print all config files that it will try to read
print(parser.files)
# reads parser files that are actually found and prints their names
print(parser.config.read(parser.files))
create_main_parser is function that creates parser which pip uses to read params from command line(optparse) and loading configs(configparser)
Possible file names for configurations are generated in get_config_files. Including PIP_CONFIG_FILE environment variable if it set.
parser.config is instance of RawConfigParser so all generated file names in get_config_files are passed to parser.config.read
.
Attempt to read and parse a list of filenames, returning a list of filenames which were successfully parsed. If filenames is a string, it is treated as a single filename. If a file named in filenames cannot be opened, that file will be ignored. This is designed so that you can specify a list of potential configuration file locations (for example, the current directory, the user’s home directory, and some system-wide directory), and all existing configuration files in the list will be read. If none of the named files exist, the ConfigParser instance will contain an empty dataset. An application which requires initial values to be loaded from a file should load the required file or files using read_file() before calling read() for any optional files:
From how I see it, your question can be interpreted in three ways:
What is the configuration of the pip executable?
There is a quite extensive documentation for the configurations supported by pip, see here: https://pip.pypa.io/en/stable/user_guide/#configuration
What is the configuration that pip uses when configuring and subsequently building code required by a Python module?
This is specified by the package that is being installed. The package maintainer is responsible for producing a configuration script. For example, Numpy has a Configuration class (https://github.com/numpy/numpy/blob/master/numpy/distutils/misc_util.py) that they use to configure their Cython build.
What are the current modules installed with pip so I can reproduce a specific environment configuration?
This is easy, pip freeze > requirements.txt. This will produce a file of all currently installed pip modules along with their exact versions. You can then do pip install -r requirements.txt to reproduce that exact environment configuration on another machine.
I hope this helps.
You can run pip in pdb. Here's an example inside ipython:
>>> import pip
>>> import pdb
>>> pdb.run("pip.main()", globals())
(Pdb) s
--Call--
> /usr/lib/python3.5/site-packages/pip/__init__.py(197)main()
-> def main(args=None):
(Pdb) b /usr/lib/python3.5/site-packages/pip/baseparser.py:146
Breakpoint 1 at /usr/lib/python3.5/site-packages/pip/baseparser.py:146
(Pdb) c
> /usr/lib/python3.5/site-packages/pip/baseparser.py(146)__init__()
-> if self.files:
(Pdb) p self.files
['/etc/xdg/pip/pip.conf', '/etc/pip.conf', '/home/andre/.pip/pip.conf', '/home/andre/.config/pip/pip.conf']
The only trick here was looking up the path of the baseparser (and knowing that the files are in there). If you don't know this already you can simply step through the program or read the source. This type of exploration should work for most Python programs.
When I create a virtualenv, it installs setuptools and pip. Is it possible to add new packages to this list?
Example use cases:
Following this solution to use ipython in virtualenv (from this question) requires installing ipython in every virtualenv (unless I allow system-site-packages).
Or if I'm doing a only flask/pygame/framework development, I'd want it in every virtualenv.
I took a different approach from what is chosen as the correct answer.
I chose I directory, like ~/.virtualenv/deps and installed packages in there by doing
pip install -U --target ~/.virtualenv/deps ...
Next in ~/.virtualenv/postmkvirtualenv I put the following:
# find directory
SITEDIR=$(virtualenvwrapper_get_site_packages_dir)
PYVER=$(virtualenvwrapper_get_python_version)
# create new .pth file with our path depending of python version
if [[ $PYVER == 3* ]];
then
echo "$HOME/.virtualenvs/deps3/" > "$SITEDIR/extra.pth";
else
echo "$HOME/.virtualenvs/deps/" > "$SITEDIR/extra.pth";
fi
Post that basically says the same thing.
You can write a python script, say personalize_venv.py that extends the EnvBuilder class and override its post_setup() method for installing any default packages that you need.
You can get the basic example from https://docs.python.org/3/library/venv.html#an-example-of-extending-envbuilder.
This doesn't need a hook. Directly run the script with command line argument dirs pointing to your venv directory/directories. The hook is the post_setup() method itself of EnvBuilder class.