Related
I have some code in a .ipynb file and got it to the point where I don't really need the "interactive" feature of IPython Notebook. I would like to just run it straight from a Mac Terminal Command Line.
Basically, if this were just a .py file, I believe I could just do python filename.py from the command line. Is there something similar for a .ipynb file?
nbconvert allows you to run notebooks with the --execute flag:
jupyter nbconvert --execute <notebook>
If you want to run a notebook and produce a new notebook, you can add --to notebook:
jupyter nbconvert --execute --to notebook <notebook>
Or if you want to replace the existing notebook with the new output:
jupyter nbconvert --execute --to notebook --inplace <notebook>
Since that's a really long command, you can use an alias:
alias nbx="jupyter nbconvert --execute --to notebook"
nbx [--inplace] <notebook>
From the command line you can convert a notebook to python with this command:
jupyter nbconvert --to python nb.ipynb
https://github.com/jupyter/nbconvert
You may have to install the python mistune package:
sudo pip install -U mistune
In your Terminal run ipython:
ipython
then locate your script and put there:
%run your_script.ipynb
You can export all your code from .ipynb and save it as a .py script. Then you can run the script in your terminal.
Hope it helps.
Using ipython:
ipython --TerminalIPythonApp.file_to_run=<notebook>.ipynb
Normally, I would prefer this option as it is really self-describing. If you prefer to use less characters, use:
ipython -c "%run <notebook>.ipynb"
which is basically what Keto already suggested (unfortunately a little bit hidden) as a comment.
In my case, the command that best suited me was:
jupyter nbconvert --execute --clear-output <notebook>.ipynb
Why? This command does not create extra files (just like a .py file) and the output of the cells is overwritten everytime the notebook is executed.
If you run:
jupyter nbconvert --help
--clear-output
Clear output of current file and save in place, overwriting the existing notebook.
For new version instead of:
ipython nbconvert --to python <YourNotebook>.ipynb
You can use jupyter instend of ipython:
jupyter nbconvert --to python <YourNotebook>.ipynb
Update with quoted comment by author for better visibility:
Author's note "This project started before Jupyter's execute API, which is now the recommended way to run notebooks from the command-line. Consider runipy deprecated and unmaintained." – Sebastian Palma
Install runipy library that allows running your code on terminal
pip install runipy
After just compiler your code:
runipy <YourNotebookName>.ipynb
You can try cronjob as well. All information is here
I had the same problem and I found papermill. The advantages against the others solutions is that you can see the results while the notebook is running. I find this feature interesting when the notebook takes very long. It is very easy to use:
pip install papermill
papermill notebook.ipynb output.ipynb
It has also, other handy options as saving the output file to Amazon S3, Google Cloud, etc. See the page for more information.
You can also use the boar package to run your notebook within a python code.
from boar.running import run_notebook
outputs = run_notebook("nb.ipynb")
If you update your notebook, you won't have to convert it again to a python file.
More information at:
https://github.com/alexandreCameron/boar/blob/master/USAGE.md
There is now a jupyter run subcommand that will execute a notebook.
jupyter run notebook.ipynb
More on this command can be found in the Jupyter documentation.
From the terminal run
jupyter nbconvert --execute --to notebook --inplace --allow-errors --ExecutePreprocessor.timeout=-1 my_nb.ipynb
The default timeout is 30 seconds. -1 removes the restriction.
If you wish to save the output notebook to a new notebook you can use the flag --output my_new_nb.ipynb
You can also use jupytext https://jupytext.readthedocs.io/en/latest/index.html.
This allows you to pair your notebook. So for each ipynb file you have a .py file as well with some comments. The .py file can be executed as usual.
You can enjoy benefits of two worlds with the cost of one extra file though.
Oh, and by the way if you are using version control you can only commit .py files with a nice diff instead of the ugly .ipynb diffs.
(The syntax in the .py files is similar to Databricks notebooks iy you are familiar with them...)
In a batch file paste the below. Use ipython to run .ipynb files.
#echo on
call "C:\ProgramData\Anaconda3\Scripts\activate.bat"
ipython "path\test.ipynb"
pause
I've made some research on this topic and wrote an article on 4 ways to run Jupyter Notebook in command line below is summary of my findings.
1. Use nbconvert
The nbconvert package is already installed in Jupyter Notebook. It can be used to execute notebook. Additionally it has many features:
can export notebook to PDF or HTML,
can hide code in output notebook,
can execute notebook even with errors in cells.
Example notebook execution:
jupyter nbconvert --execute --to notebook --allow-errors your-notebook.ipynb
The above command will output your-notebook.nbconvert.ipynb file and will execute all cells even with errors.
2. Use papermill
The papermill allows you to parametrize notebook. You can define variables as parameters (with cell tags).
Example command:
papermill -p name Piotrek your-notebook.ipynb output-notebook.ipynb
Example notebook with injected parameters:
3. Manually download notebook as .py script
There is an option to manually download notebook as .py script:
After download you can add execution rights to the file and run it as a command line script.
4. Use jupytext
The jupytext package allows you to synchronize .ipynb file with .py file. You don't need to manually convert notebook to script.
I have some Python code in a Jupyter notebook and I need to run it automatically every day, so I would like to know if there is a way to set this up. I really appreciate any advice on this.
Update
recently I came across papermill which is for executing and parameterizing notebooks.
https://github.com/nteract/papermill
papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1
This seems better than nbconvert, because you can use parameters. You still have to trigger this command with a scheduler. Below is an example with cron on Ubuntu.
Old Answer
nbconvert --execute
can execute a jupyter notebook, this embedded into a cronjob will do what you want.
Example setup on Ubuntu:
Create yourscript.sh with the following content:
/opt/anaconda/envs/yourenv/bin/jupyter nbconvert \
--execute \
--to notebook /path/to/yournotebook.ipynb \
--output /path/to/yournotebook-output.ipynb
You have more options except --to notebook. I like this option since you have a fully executable "log"-File afterwards.
I recommend using a virtual environment to run your notebook, to avoid that future updates mess with your script. Do not forget to install nbconvert into the environment.
Now create a cronjob, that runs every day e.g. at 5:10 AM, by typing crontab -e in your terminal and add this line:
10 5 * * * /path/to/yourscript.sh
Try the SeekWell Chrome Extension. It lets you schedule notebooks to run weekly, daily, hourly or every 5 minutes, right from Jupyter Notebooks. You can also send DataFrames directly to Sheets or Slack if you like.
Here's a demo video, and there is more info in the Chrome Web Store link above as well.
**Disclosure: I'm a SeekWell co-founder
It's better to combine with airflow if you want to have higher quality.
I packaged them in a docker image, https://github.com/michaelchanwahyan/datalab.
It is done by modifing an open source package nbparameterize and integrating the passing arguments such as execution_date. Graph can be generated on the fly The output can be updated and saved within inside the notebook.
When it is executed
the notebook will be read and inject the parameters
the notebook is executed and the output will overwrite the original path
Besides, it also installed and configured common tools such as spark, keras, tensorflow, etc.
you can add jupyter notebook in cronjob
0 * * * * /home/ec2-user/anaconda3/bin/python /home/ec2-user/anaconda3/bin/jupyter-notebook
you have to replace /home/ec2-user/anaconda3 with your anaconda install location, and you can schedule time based on your requirements in cron
Executing Jupyter notebooks with parameters is conveniently done with Papermill. I also find convenient to share/version control the notebook either as a Markdown file or a Python script with Jupytext. Then I convert the notebook to an HTML file with nbconvert. Typically my workflow looks like this:
cat world_facts.md \
| jupytext --from md --to ipynb --set-kernel - \
| papermill -p year 2017 \
| jupyter nbconvert --no-input --stdin --output world_facts_2017_report.html
Learn more about the above, including how to specify the Python environment in which the notebook is expected to run, and how to use continuous integration on notebooks, have a look at my article Automated reports with Jupyter Notebooks (using Jupytext and Papermill) which you can read either on Medium, GitHub, or on Binder. Use the Binder link if you want to test interactively the outcome of the commands in the article.
As others have mentioned, papermill is the way to go. Papermill is just nbconvert with a few extra features.
If you want to handle a workflow of multiple notebooks that depend on one another, you can try Airflow's integration with papermill. If you are looking for something simpler that does not need a scheduler to run, you can try ploomber which also integrates with papermill (Disclaimer: I'm the author).
To run your notebook manually:
jupyter nbconvert --to notebook --execute /home/username/scripts/mynotebook.ipynb
Create a simple batch file and add the command above to the file:
/home/username/scripts/mynotebook.sh
Paste the command above into the file
Make the file executable
chmod +x /home/username/scripts/mynotebook.sh
To schedule your notebook use cron or airflow, depends on your needs vs complexity. if you want to use cron, you can simply do crontab -e and add an entry
00 11 * * * /home/username/scripts/mynotebook.sh
You can download the notebook in the form of .py and then create a batch file to execute the .py script. Then schedule the batch file in the task scheduler
Creating a BAT file then running it through Task scheduler worked for me. Below is the code.
call C:\Users\...user...\Anaconda3\condabin\conda activate
python -m notebook_file.py
pause
call conda deactivate
There are several ways to execute a Jupyter Notebook daily, according to the article.
Cron or Windows Task Scheduler
You can use your operating system scheduler to execute the notebook. There are two command line tools for executing notebooks:
nbconvert
papermill
Both are great, I personally use nbconvert, but papermill offers handful of extensions as input parameters for notebooks or automatic export to cloud storage.
Mercury
The open source framework Mercury is a web based application that:
can execute notebook in the background,
can share notebook as website,
can send execute notebook as email with PDF or HTML attachment,
can restrict access to notebooks to authenticated users.
Notebooks available in web app
Scheduled notebook
PDF notebook sent in email
Notebooker
Notebooker is open source web app for scheduling and sharing notebooks.
List of notebooks
Executed notebook
You want to use Google AI Platform Notebooks Scheduler service currently in EAP.
I want to execute one python script in Jupyter, but I don't want to use the web browser (IPython Interactive terminal), I want to run a single command in the Linux terminal to load & run the python script, so that I can get the output from Jupyter.
I tried to run jupyter notebook %run <my_script.py>, but it seems jupyter can't recognize %run variable.
Is it possible to do that?
The jupyter_client package has a jupyter run command since v5.0, so you can just do:
jupyter run my_script.py
You can use the jupyter console -i command to run an interactive jupyter session in your terminal. From there you can run import <my_script.py>. Do note that this is not the intended use case of either jupyter or the notebook environment. You should run scripts using your normal python interpreter instead.
You can run this command to run an interactive jupyter session in your terminal.
jupyter notebook
I have some code in a .ipynb file and got it to the point where I don't really need the "interactive" feature of IPython Notebook. I would like to just run it straight from a Mac Terminal Command Line.
Basically, if this were just a .py file, I believe I could just do python filename.py from the command line. Is there something similar for a .ipynb file?
nbconvert allows you to run notebooks with the --execute flag:
jupyter nbconvert --execute <notebook>
If you want to run a notebook and produce a new notebook, you can add --to notebook:
jupyter nbconvert --execute --to notebook <notebook>
Or if you want to replace the existing notebook with the new output:
jupyter nbconvert --execute --to notebook --inplace <notebook>
Since that's a really long command, you can use an alias:
alias nbx="jupyter nbconvert --execute --to notebook"
nbx [--inplace] <notebook>
From the command line you can convert a notebook to python with this command:
jupyter nbconvert --to python nb.ipynb
https://github.com/jupyter/nbconvert
You may have to install the python mistune package:
sudo pip install -U mistune
In your Terminal run ipython:
ipython
then locate your script and put there:
%run your_script.ipynb
You can export all your code from .ipynb and save it as a .py script. Then you can run the script in your terminal.
Hope it helps.
Using ipython:
ipython --TerminalIPythonApp.file_to_run=<notebook>.ipynb
Normally, I would prefer this option as it is really self-describing. If you prefer to use less characters, use:
ipython -c "%run <notebook>.ipynb"
which is basically what Keto already suggested (unfortunately a little bit hidden) as a comment.
In my case, the command that best suited me was:
jupyter nbconvert --execute --clear-output <notebook>.ipynb
Why? This command does not create extra files (just like a .py file) and the output of the cells is overwritten everytime the notebook is executed.
If you run:
jupyter nbconvert --help
--clear-output
Clear output of current file and save in place, overwriting the existing notebook.
For new version instead of:
ipython nbconvert --to python <YourNotebook>.ipynb
You can use jupyter instend of ipython:
jupyter nbconvert --to python <YourNotebook>.ipynb
Update with quoted comment by author for better visibility:
Author's note "This project started before Jupyter's execute API, which is now the recommended way to run notebooks from the command-line. Consider runipy deprecated and unmaintained." – Sebastian Palma
Install runipy library that allows running your code on terminal
pip install runipy
After just compiler your code:
runipy <YourNotebookName>.ipynb
You can try cronjob as well. All information is here
I had the same problem and I found papermill. The advantages against the others solutions is that you can see the results while the notebook is running. I find this feature interesting when the notebook takes very long. It is very easy to use:
pip install papermill
papermill notebook.ipynb output.ipynb
It has also, other handy options as saving the output file to Amazon S3, Google Cloud, etc. See the page for more information.
You can also use the boar package to run your notebook within a python code.
from boar.running import run_notebook
outputs = run_notebook("nb.ipynb")
If you update your notebook, you won't have to convert it again to a python file.
More information at:
https://github.com/alexandreCameron/boar/blob/master/USAGE.md
There is now a jupyter run subcommand that will execute a notebook.
jupyter run notebook.ipynb
More on this command can be found in the Jupyter documentation.
From the terminal run
jupyter nbconvert --execute --to notebook --inplace --allow-errors --ExecutePreprocessor.timeout=-1 my_nb.ipynb
The default timeout is 30 seconds. -1 removes the restriction.
If you wish to save the output notebook to a new notebook you can use the flag --output my_new_nb.ipynb
You can also use jupytext https://jupytext.readthedocs.io/en/latest/index.html.
This allows you to pair your notebook. So for each ipynb file you have a .py file as well with some comments. The .py file can be executed as usual.
You can enjoy benefits of two worlds with the cost of one extra file though.
Oh, and by the way if you are using version control you can only commit .py files with a nice diff instead of the ugly .ipynb diffs.
(The syntax in the .py files is similar to Databricks notebooks iy you are familiar with them...)
In a batch file paste the below. Use ipython to run .ipynb files.
#echo on
call "C:\ProgramData\Anaconda3\Scripts\activate.bat"
ipython "path\test.ipynb"
pause
I've made some research on this topic and wrote an article on 4 ways to run Jupyter Notebook in command line below is summary of my findings.
1. Use nbconvert
The nbconvert package is already installed in Jupyter Notebook. It can be used to execute notebook. Additionally it has many features:
can export notebook to PDF or HTML,
can hide code in output notebook,
can execute notebook even with errors in cells.
Example notebook execution:
jupyter nbconvert --execute --to notebook --allow-errors your-notebook.ipynb
The above command will output your-notebook.nbconvert.ipynb file and will execute all cells even with errors.
2. Use papermill
The papermill allows you to parametrize notebook. You can define variables as parameters (with cell tags).
Example command:
papermill -p name Piotrek your-notebook.ipynb output-notebook.ipynb
Example notebook with injected parameters:
3. Manually download notebook as .py script
There is an option to manually download notebook as .py script:
After download you can add execution rights to the file and run it as a command line script.
4. Use jupytext
The jupytext package allows you to synchronize .ipynb file with .py file. You don't need to manually convert notebook to script.
I am deploying a Python package, and I would like to run a simple test to see if all cells in my notebook will run without errors. I would like to test this via commandline as I have issues running a notebook in virtualenv. Is there are simple command-line way to test this?
Note to the moderator: this question has been marked as a duplicate of How to run an .ipynb Jupyter Notebook from terminal?
. However, this question was posted (asked Feb 18 '16 at 2:49) several days before that one (asked Feb 22 '16 at 3:35). At most, that post might be marked as a duplicate, and if deemed so, an appropriate action would be to merge the two questions, maintaining this, the original, as the master.
However, these questions may not be duplicates (the intent of the other author is unclear). Regardless, this question and it's answers specifically address executing cells within a jupyter notebook from the terminal, not simply converting notebooks to python files.
nbconvert (a jupyter tool for notebook conversion) allows you to do this without any extra packages:
Just go to your terminal and type
$ jupyter nbconvert --to notebook --inplace --execute mynotebook.ipynb
Source
(Thanks Stephan for suggesting the --inplace flag)
NOTE: This said, I'd try to convert everything you need to a proper script. Jupyter notebooks are thought to use for exploring and sharing results, and not as a replacement of traditional programs.
You can use runipy to do this.
runipy will run all cells in a notebook. If an error occurs, the process will stop.
$ pip install runipy
$ runipy MyNotebook.ipynb
There are also commands for saving the output file as a notebook or an html report:
$ runipy MyNotebook.ipynb OutputNotebook.ipynb
$ runipy MyNotebook.ipynb --html report.html
You can also try papermill which allows you to execute notebooks from command line, and also pass parameters:
For example:
$ papermill mynotebook.ipynb mynotebook_output.ipynb -p start "2017-11-01" -p end "2017-11-30"
You can also run it without passing any parameter.
You could use nbconvert to convert the ipynb file into a python script and then run it.
jupyter nbconvert --to python notebook.ipynb
python3 notebook.py