I have Anaconda Navigator on my work computer and I've changed the default working directory for Jupyter notebooks to be a certain location on the firm server, using the steps given here
I have also created a second environment in Anaconda, for which I would like to use a different Jupyter Notebook working directory than that of the base (root). To do this, I believe I would need to:
Create a second Jupyter Notebook config file
Get the second environment to refer to the new config file, while ensuring that the old file still referred to the original config file.
How would I go about this? Alternate approaches to creating multiple working directories also welcome.
Was looking to achieve the same result: pass different configurations which would include specific settings such as working and workspace directories when launching Jupyter Notebook/Lab in different conda or other virtual environments.
Noticing the following:
jupyter lab --help
--config=<Unicode>
Full path of a config file.
Default: ''
Equivalent to: [--JupyterApp.config_file]
Thus, to achieve the desired result, it is possible to pass the path to the relevant config file upon launching Jupyter Lab/Notebook as such:
jupyter lab --config=~/.jupyter/path_to_my_custom_jupyter_config_for_env_1.py
# or
jupyter notebook --config=~/.jupyter/path_to_my_custom_jupyter_config_for_env_1.py
You can copy your current configuration file, make the relevant adjustments and save it as a different file. You would then pass the path to that new config file when launching Jupyter as per above. To simplify, you can create shortcuts that do that (instead of typing/copying the specific parameters on each launch).
Otherwise (with no specific argument passed), Jupyter Lab/Notebook will launch using the default configuration file if it exists, which is always located in the home directory of the Unix user launching Jupyter (~/.jupyter/jupyter_notebook_config.py).
Related
Desired behaviour
We have an existing workflow in vanilla Jupyter Notebook/Lab where we use relative paths to store outputs of some notebooks. Example:
/home/user/notebooks/notebook1.ipynb
/home/user/notebooks/notebook1_output.log
/home/user/notebooks/project1/project.ipynb
/home/user/notebooks/project1/project_output.log
In both notebooks, we produce the output by simply writing to ./output.log or so.
Problem
However, we are now trying Google Dataproc with Jupyter optional component, and the current directory is always / regardless of which notebook it's run from. This applies for both the notebook and Lab interfaces.
What I've tried
Disabling c.FileContentsManager.root_dir='/' in /etc/jupyter/jupyter_notebook_config.py causes the current directory to be set to wherever I started jupyter notebook from, but it is always that initial starting folder instead of following the .ipynb notebook files.
Any idea on how to restore the "dynamic" current directory behaviour?
Even if it's not possible, I'd like to understand how Dataproc even makes Jupyter behave differently.
Details
Dataproc Image 2.0-debian10
Notebook Server 6.2.0
Jupyterlab 3.0.18
No it is not possible to always get the current directory where your .ipynb file is. Jupyter is running from the local filesystem of the master node of your cluster. It will always take the default system path for its kernel.
In other cases(besides dataproc) also it is not possible to consistently get the path of a Jupyter notebook. You can check out this thread regarding this topic.
You have to mention the directory path for your log file to be saved in the desired path.
Note that the GCS folder in your Lab refers to the Google Cloud storage Bucket of your cluster. You can create .ipynb in GCS but when you will execute the file it will be running inside the local system.Thus you will not be able to save log files in GCS directly.
EDIT:
It's not only Dataproc who makes Jupyter behave differently.If you use Google Colab notebooks there you will also see the same behaviour.
The reason is because youre always executing code in the kernel does not matter where the file is. And in theory multiple notebooks could connect to that kernel.Thus you can't have multiple working directories for the same kernel.
As I mentioned earlier by default if you're starting a notebook, the current working directory is set to the path of the notebook.
Link to the main thread -> https://github.com/ipython/ipython/issues/10123
Definitely a general solution for most use-cases seems to be what is described right here in the github issue: https://github.com/ipython/ipython/issues/10123#issuecomment-354889020
I have two different project folders. I want to make one of the projects a root so that when I import something from first project in the second project it does not show ModuleNotFoundError. For example, In PyCharm we can set project dependencies. Is there way to do so in Jupyter Notebook?
The best way to handle this in notebooks or jupyterlab would be to set the PYTHONPATH environment variable. For example, on MAC or unix-based OS, using bash, do this:
export PYTHONPATH="${PYTHONPATH}:/path/to/project"
Then re-load the notebook. Adding your project path to the PYTHONPATH variable will allow you to import any modules that are specified in this variable.
Additionally, you can set this variable in a jupyter notebook itself by entering the bash command into a notebook cell, prepended with the ! symbol like this:
! export PYTHONPATH="${PYTHONPATH}:/path/to/project"
Then executing the cell.
I have experience in RStudio and Visual Studio and new to Python environment, I am struggling to set Python project in Jupyter like above two IDE. Here when I open new notebook then it create it as default location and I am not finding a way to put the file in my project location, Just want to know how to make production ready project like folder structure in Jupyter.
If this post is duplicate then I am not able to search the original one.
Under Linux, specify which directory Jupyter notebook uses to locate the notebooks with the command:
jupyter notebook --notebook-dir=<path to notebook location>
I have this aliased in my .bashrc file to jupytern.
In Windows I created a desktop shortcut and specified the notebook directory as:
C:\python36\Scripts\jupyter-notebook.exe --notebook-dir=<path to notebook locations>
You need to change the path to the jupyter-notebook.exe file to where it is on your installation.
I am trying to use the guidance on http://www.slideshare.net/fullscreen/randyzwitch/ipython-ec2/12 to install public Ipython notebooks in an AWS instance. One problem that i encounter is, when i try to create a profile, i do not not observe the creation of an ipython_notebook_config_py file (as explained in the tutoral, as per the screenshot), but i only get a ipython_kernel_config.py file, which has very different contents, and cannot be edited in the way as explained in the tutoral. Can someone help me to understand why this happens, and what i should do subsequently? Many thanks.
The notebook server is no longer part of IPython; it's a separate Jupyter project, which has its own config directory ~/.jupyter. The config file for the notebook server is ~/.jupyter/jupyter_notebook_config.py. You can use this to configure the notebook server. If you want to keep multiple configurations of the notebook server, you can use the environment variable JUPYTER_CONFIG_DIR to specify that a different directory should be used:
JUPYTER_CONFIG_DIR=~/jupyter_nbserver jupyter notebook
Note: IPython has not lost its profiles or config files, so ~/.ipython/profile_default/ipython_config.py and startup files, etc. continue to work as before for configuring the IPython kernel, just not the notebook server itself.
References:
Running a public notebook server
Jupyter configuration
Migration from IPython to Jupyter
There was official(?) recommendation of running an IPython Notebook server, and creating a profile via
$ ipython profile create nbserver
as recommended in http://ipython.org/ipython-doc/1/interactive/public_server.html. This allowed for very different and very useful behavior when starting an IPython Notebook via ipython notebook and ipython notebook --profile=nbserver.
With Jupyter 4.0, there's a change and there are no longer profiles. I've found the conversation https://gitter.im/ipython/ipython/archives/2015/05/29 which has user minrk saying:
The .ipython directory has several things in it:
multiple config directories (called profiles)
one 'data' directory, containing things like kernelspecs, nbextensions
runtime info scattered throughout, but mostly in profiles
Jupyter follows more platform-appropriate conventions:
one config dir at JUPYTER_CONFIG_DIR, default: .jupyter
one data dir at JUPYTER_DATA_DIR, default: platform-specific
one runtime dir at JUPYTER_RUNTIME_DIR, default: platform-specific
And a rather cryptic remark:
If you want to use different config, specify a different config directory with JUPYTER_CONFIG_DIR=whatever
What's the best way to get different behavior (say, between when running as a server vs normal usage)?
Will it involve running something like:
$ export JUPYTER_CONFIG_DIR=~/.jupyter-nbserver
$ jupyter notebook
whenever a server 'profile' needs to be run? and
$ export JUPYTER_CONFIG_DIR=~/.jupyter
$ jupyter notebook
whenever a 'normal' profile needs to run? Because that seems terrible. What's the best way to do this in Jupyter 4.0?
Using some code from this blog post http://www.svds.com/jupyter-notebook-best-practices-for-data-science/ and updating it. The easiest solution appears to be to create an alias, like:
alias jupyter-nbserver='JUPYTER_CONFIG_DIR=~/.jupyter-nbserver jupyter notebook'
So now you can run the jupyter notebook with a different config via the simple command jupyter-nbserver.
A more robust solution might involve creating a bash function that changes the environment variable, checks whether there's a config file, if not creating one, then executing, but that's probably overkill. The answer that I give on this related question https://stackoverflow.com/a/32516200/246856 goes into creating the initial config files for a new 'profile'.