I'm trying to get a basic ipyparallel environment working using mpi4py as described in the ipyparallel documentation. After starting the ipcluster, I load ipython and try to create a client but it has no IDs and accessing the directview returns a NoEnginesRegistered exception.
Steps I take to get to this point:
Create a new environment: conda create --name=ipyparallel and source activate ipyparallel
Install ipyparallel and mpi4py: conda install ipyparallel mpi4py
Create a new ipython profile: ipython profile create --parallel --profile=mpi
Edit ~/.ipython/profile_mpi/ipcluster_config.py and add c.IPClusterEngines.engine_launcher_class = 'MPIEngineSetLauncher'
Launch cluster with ipcluster start --profile=mpi
Then I launch ipython and run the following:
import ipyparallel as ipp
c = ipp.client(profile="mpi")
c[:] # <-- NoEnginesRegistered exception
Step 5 reports that "Engines appear to have started successfully" and I can see that a process named "mpiexec" is running. Strangely, I tried these same steps on another machine with the same OS and there it worked with no problems. What am I missing?
I solved the problem for anyone coming here with a similar problem. During installation I had added the notebook extension to jupyter's global config. Not sure why that caused this problem but it's fixed now. Outside of a conda environment, I ran:
sudo pip install ipyparallel
sudo jupyter nbextension disable --py ipyparallel
sudo jupyter nbextension uninstall --py ipyparallel
sudo pip uninstall ipyparallel
and then inside the conda environment I can connect to the ipyparallel engines.
Related
I am not able to use virtual environments in JupyterHub version 3.0.14. I have no shell access to the server, but I can use the JupyterHub terminal.
step 1
After reading some documentation, I did this:
installed ipykernel in the base environment with: pip install --user ipykernel in the JupyterHub terminal.
created a new environment with python -m venv test-env
created a new environment with python -m venv test-env2
run python -m ipykernel install --user --name=test-env
run python -m ipykernel install --user --name=test-env2
After the previous sub-steps, I find the new environments in JupyterHub and I can open a new notebook using the menu "new"->"name of the environment".
step 2
In JupyterHub, I select "new" -> "test-env". In the new notebook, I try and import a new module, for example, hiplot, receiving an error because it is not installed.
To install hiplot in the test-env environment, I open a JupyterHub terminal, activate the environment with source [...]/bin/activate, then pip install hiplot, with success. If I run python from the command line and import hiplot I receive no errors.
step 3
I create a new notebook in the test-env as in step 2 ("new"->"test-env" from the menu), but I still cannot use hiplot, that results not installed. If I try and import it, JupyterHub tells me it is not installed.
QUESTION: After steps 2 and 3, I assume that test-env is not referring to the same environment when accessed from the command line or JupyterHub. Am I correct?
step 4
Inside of a notebook running within the test-env in JupyterHub, I run: `
import sys
!{sys.executable} -m pip install hiplot
After the previous command, I can import hiplot in the current notebook.
BUT
step 5
I open a new JupyterHub notebook in the base or test-env2 environments, where I thought hiplot was not be available. But I was wrong: the module hiplot is available in both of them.
It seems that, in step 4, hiplot has been installed in the base environment and is thus available in all the JupyterHub environments.
I tried to repeat the same steps changing step 1: instead of installing ipykernel in the base environment, I installed it in the test-env, with no success.
QUESTIONS: Is it really possible to use environments in JupyterHub? If it is, what am I doing wrong? I kept reading help pages, but I keep finding instructions with the same commands I used...
QUESTION how can I share a set of installed modules between the system python and the Jupyter notebooks?
The best way to achive what you want is by using conda.
Have a look to this video LInkAnaconda for Python where he explains quite well some differences when using pip install or conda install for your vens
I have got two issues with modin
Lately I am not able to create modin environment using the command:
conda create -n aikit-modin intel-aikit-modin -c intel -c conda-forge
(as mentioned in the documentation link of AI Analytics toolkit https://software.intel.com/content/www/us/en/develop/articles/installing-ai-kit-with-conda.html#gs.2irfv6 )
It seems documentation correction is required.
I was able to create modin environment using stock
conda create -n stock-modin modin -c conda-forge
(referred-https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/distribution-of-modin.html#gs.2isi2d and Data Science at Scale with Modin documentation:
https://medium.com/intel-analytics-software/data-science-at-scale-with-modin-5319175e6b9a )
But while trying to do the modin sample, NYC_Taxi.ipynb, in the online JupyterLab of Intel devcloud, I am getting a
file save server error JupyterHub server no longer running at
/user/uXXXXX/api/contents/modin/examples/jupyter/NYC_Taxi.ipynb.
Restart the server at
http://jupyter.oneapi.devcloud.intel.com/hub/spawn/uXXXXX
as shown in the screenshot. I was not able to complete the notebook. It seems like a kernel error, even though I created a fresh kernel from the stock-modin environment.
Screenshot
(source of the NYC sample:https://github.com/modin-project/modin/tree/master/examples )
How to execute the samples of Intel Distribution of Modin? Any workarounds?
conda create -n aikit-modin intel-aikit-modin -c intel -c conda-forge
This command takes time to execute. As you can create modin environment using stock, you can execute the notebook using the below method.
As a workaround, devcloud tunneling is a solution to this kernel error. Open terminals like Cygwin, vscode, mobaxterm or openssh. Setting up of devcloud connection should be done before going to the steps below. Please refer Connect to the Devcloud option in https://devcloud.intel.com/oneapi/documentation/shell-commands/
If you have connected to devcloud using any of the above terminals, please follow the below steps.
In the first terminal execute the following commands
ssh devcloud //(To connect to devcloud)
qsub -I //(to request compute node)
copy the compute node //(it would be of this format sXXX-nXXX)
conda create -n stock-modin modin -c conda-forge //(to create modin environment using stock)
source activate stock-modin //(activate modin environment)
conda install pip
conda install ipykernel //(package for creating kernels)
python -m ipykernel install --user --name myenv --display-name "Python (myenv)" //(command to create kernels)
jupyter notebook --no-browser --port=<desired_port_number> //(to launch Jupyter)
Copy any one of the URLs
“Or copy and paste one of these URLs:
http://localhost: <desired_port_number>/?token=2deea502bd258ed648f59eb11383bff4fd0edf5a4ce4f4f3or
http://127.0.0.1: <desired_port_number>/?token=2deea502bd258ed648f59eb11383bff4fd0edf5a4ce4f4f3”
Run the below commands in another terminal: (Keep the first terminal open)
ssh -L <desired_port_number>:localhost: <desired_port_number> devcloud
ssh -L <desired_port_number>:localhost: <desired_port_number> <compute node you copied>
Open the copied URL in a browser
You’ll get Jupyter, open the required notebook.
Select the kernel that you created
Execute the notebook.
I'm not able to find a package from my notebook when I have installed it via pip / conda in my terminal.
For example, I did pip install trading-calendars and conda install trading-calendars in terminal but from trading_calendars import get_calendar in Jupyter notebook throws a ModuleNotFoundError with the message No module named 'trading_calendars'.
Is it possible that Jupyter takes time to refresh? If this keeps happening despite restarting Jupyter notebook, what should I do?
Duplicates:
How to list imported modules?
Package for listing version of packages used in a Jupyter notebook
Possible solution:
sys.path different in Jupyter and Python - how to import own modules in Jupyter?
This can be pretty confusing with Jupyter. It's very important to realize that your Jupyter client can connect to different "kernels" which equates to various python environments you might have installed. That is, you can start the Jupyter server with one python environment, and be executing your notebook's cells from another.
You need to make sure that you have the libraries installed to the environment that your kernel is using.
You will need to generate a kernelspec for your environment if you haven't already.
You can create a kernelspec using ipykernel. Here's an example of me doing it with conda.
$ conda activate test
$ conda install ipykernel
$ python -m ipykernel install --user --name test \
--display-name "Python (test)"
You can view your kernelspecs with this command
{~/path/to/project} (master *$)$ jupyter kernelspec list
Available kernels:
django_extensions /Users/nicholasbrady/Library/Jupyter/kernels/django_extensions
python3 /Users/nicholasbrady/anaconda3/share/jupyter/kernels/python3
python2 /usr/local/share/jupyter/kernels/python2
I am currently trying to work basic python - jupyter projects.
I am stuck on following error during matplotlib:
screenshot on jupyter-error
ModuleNotFoundError: No module named 'matplotlib'
I tried to update, reinstall matplotlib aswell in conda and in pip but it still not working.
happy over every constructive feedback
In a Notebook's cell type and execute the code:
import sys
!{sys.executable} -m pip install --user matplotlib
and reload the kernel
(src: http://jakevdp.github.io/blog/2017/12/05/installing-python-packages-from-jupyter/ )
open terminal and change the directory to Scripts folder where python installed. Then type the following command and hit enter
pip install matplotlib
Hope this will solve the issue.
I was facing the exact issue. It turns out that it was using the system Python version despite me having activated my virtual environment.
This is what eventually worked.
If you are using a virtual environment which has a name say myvenv, first activate it using command:
source activate myvenv
Then install module ipykernel using the command:
pip install ipykernel
Finally run (change myvenv in code below to the name of your environment):
ipykernel install --user --name myvenv --display-name "Python (myvenv)"
Now restart the notebook and it should pick up the Python version on your virtual environment.
While #Frederic's top-voted solution is based on JakeVDP's blog post from 2017, it completely neglects the %pip magic command mentioned in the blog post. Since 2017, that has landed in mainline IPython and the easiest way to access the correct pip instance connected to your current IPython kernel and environment from within a Jupyter notebook is to do
%pip install matplotlib
Take a look at the list of currently available magic commands at IPython's docs.
generally speaking you should try to work within python virtual environments. and once you do that, you then need to tell JupyterLab about it. for example:
# create a virtual environment
# use the exact python you want to work with in this step
python3.9 -m venv myvenv
# 'activate' (or 'enter') it
source myvenv/bin/activate
# install the exact stuff you want to use in that environment
pip install matplotlib
# now tell JupyterLabs about the environment
python -m ipykernel install --user --name="myenv" --display-name="My project (myenv)"
# start it up
jupyter notebook mynotebook
# if you now look under 'Kernel->Change kernel', your 'myenv' should be there
# select it (restart kernel etc if needed) and you should be good
The issue with me was that jupyter was taking python3 for me, you can always check the version of python jupyter is running on by looking on the top right corner (attached screenshot).
When I was doing pip install it was installing the dependencies for python 2.7 which is installed on mac by default.
It got solved by doing:
> pip3 install matplotlib
Having the same issue, installing matplotlib before to create the virtualenv solved it for me. Then I created the virtual environment and installed matplotlib on it before to start jupyter notebook.
in jupter notebook type
print(sys.executable)
this gave me the following
/Users/myusername/opt/anaconda3/bin/python
open terminal, go into the folder
/Users/myusername/opt/anaconda3/bin/
type the following:
python3 -m pip install matplotlib
restart jupyter notebook (mine is vs code mac ox)
If module installed an you are still getting this error, you might need to run specific jupyter:
python -m jupyter notebook
and this is also works
sudo jupyter notebook --allow-root
I am new to jupyter/ipython. I usually start the notebook on a remote machine and create a ssh tunnel. Details of how the tunnel is set up can be found here: http://www.hydro.washington.edu/~jhamman/hydro-logic/blog/2013/10/04/pybook-remote/
However, many times when I start a notebook, it reports back with 0 active kernels (not all the times). Following is the screenshot of that:
-bash-4.1$ jupyter notebook --no-browser --port=7777
[NotebookApp] Serving notebooks from local directory: /x/y/z
[NotebookApp] 0 active kernels
[NotebookApp] The Jupyter Notebook is running at: http://localhost:7777/
[NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
I obviously have python.
python --version
Python 2.7.11 :: Anaconda 2.0.1 (64-bit)
Can someone explain to me this erratic behavior? and how it can be fixed?
Any help would be appreciated.
Thank you
Attached is the error I get:
I also use jupyter using a ssh tunnel and was getting 0 active kernels message and I could not run any notebook. My error snapshot is here. The ipython kernel was not installed properly in the direct install of jupyter using sudo -H pip install jupyter. I resolved it by updating the ipython kernel using:
sudo apt-get -y install ipython ipython-notebook
sudo -H pip install jupyter
If it doesn't work, try uninstalling jupyter and then install packages in sequence as explained in this link.