I am unable to upgrade my Python version to 3.9 on Vertex AI on GCP - I tried the methods outlined in several questions but they did not make for it - is there anyone who can help with this ?
Thanks
I tried updating the environment variables and installing Python 3.9 using the terminal but that did not seem to work.
The notebooks use Python 3.7 however, there are 2 alternatives to use python 3.9 in your Vertex AI user-managed notebooks:
Method 1:
You can use a custom image when creating the notebook. This way you can have the python version you prefer installed, as well as any dependencies you need preinstalled. On the creation view select the custom container option in “environment” so you can enter the route. You can find a guide with all the steps detailed here.
In the case you don’t have a custom container image created yet, you can find a guide on how to build your container image here, the guide contains explanations on using yaml or json.
Method 2:
A more straightforward solution would be to use a Conda environment with the desired Python version and use that environment as a kernel in any Jupyter notebooks.
You can follow the below steps to create a Conda environment and a kernel from it. If you are using the terminal inside the Vertex AI notebook, you will need to activate the environment before executing any commands by using command conda activate my-env the environment name shout appears in brackets in the command prompt (my-env) jupyter#jupyter:~$.
# 1. Open the terminal in the JupyterLab environment
# 2. Create a new conda env called “python39” (you can change it if you want):
$ conda create -n python39 python=3.9 -y
# 3.Activate your new Python 3.9 environment:
$ conda activate python39
# 4. Install ipykernel when logged in the new env:
(python39)$ conda install ipykernel
# 5. Install jupyter
pip install jupyter
# 6. Create kernel with name "python39-kernel":
ipython kernel install --name "python39-kernel" --user
# 7. List kernels
jupyter kernelspec list
Select the kernel named python39-kernel and check the version of python. It will show 3.9.
This is an easy workaround but it might imply some issues on a long term basis, currently I believe there is no way to change the base Python version other than using custom containers.
Related
I am having problems installing modules and then importing them into specific Jupyter Notebook kernels. I want to install them directly into the kernel as opposed to throughout anaconda to separate dependencies in projects. Here is how the problem goes:
I firstly want a package, for example, nltk
I navigate to and activate the conda environment (called python3) and run 'conda install nltk'
I then load that environment into Jupyter using ipykernel with the command 'python -m ipykernel install --user --name python3'
When trying to import the package into the notebook it tells me that it cannot be found
I have been struggling with this for a while. Where am I going wrong? I greatly appreciate all the help.
NOTE: I have somehow managed to install and import many packages into notebooks using the aforementioned process. I'd really like a method to do this in a foolproof manner.
Not entirely clear where things go wrong, but perhaps clarifying some of the terminology could help:
"navigate to...the conda environment" - navigating has zero effect on anything. Most end-users should never enter or directly write to any environment directories.
"...and activate the conda environment" - activation is unnecessary - a more robust installation command is always to use a -n,--name argument:
conda install -n python3 nltk
This is more robust because it is not context-sensitive, i.e., it doesn't matter what (if any) environment is currently activated.
"load that environment into Jupyter using ipykernel" - that command registers the environment as a kernel at a user-level. That only ever needs to be run once per kernel - not after each new package installation. Loading the kernel happens when you are creating (or changing the settings of) a notebook. That is, you choose the kernel in the Jupyter GUI.
Even better, keep jupyter in a dedicated environment with an installation of nb_conda_kernels and Jupyter (launched from that dedicated environment) will auto-discover all Conda environments that have valid kernels installed (e.g., ipykernel, r-irkernel).
I have python 3.10 installed on my PC but every time I open a new Jupyter Notebook and check the version it still says that I am using the previous version that I had. Is there a way to make it start using the latest version every time I open a new notebook?
You should try to work with virtual environments. They are very commonly used in Python. Then, on your Jupyter Notebook, you will be able to choose the particular environment (and Python version) you want to run your notebook.
First, create a virtual environment using Python 3.10:
pip install virtualenv
virtualenv nameofthevenv --python python3.10
source nameofthevenv/bin/activate
Then, inside the virtualenv:
(nameofthevenv) $ pip install jupyter ipykernel
(nameofthevenv) $ python -m ipykernel install --user --name nameofthevenv
(nameofthevenv) $ jupyter notebook
Finally, go to the page in your browser the last command opened, and choose the running kernel at the top right: Kernel >> Change Kernel >> <list of kernels>. You will see nameofthevenv. Select it and you're good to go: the python version will be Python 3.10.
I currently use this command to create a new anaconda environment:
conda create --name=<myEnvNameHere> python=3.9.5
I determined that 3.9.5 is the latest version of python available in anaconda with this command:
conda search -f python
How can I use just one command to create a new environment with the latest available version of python, perhaps something like:
conda create --name=<myEnvNameHere> python=latest
I know I can write a script to achieve the outcome I am after, but is there a way to do it in the conda create command natively?
According to the documentation for conda install, conda will (when no version is specified, it seems) try to install the latest version of the specified package(s).
Hence, you should be able to get the latest version of python in your new environment by running a command like this.
conda create --name=<myEnvNameHere> python
I have a conda environment containing all packages for jupyter notebook (say it's called jupyter_env. In a different conda environment I have R installed including r-irkernel (say the env is called R_env).
For python kernels I can easily make a python kernel in a specific environment (called e.g. pyth27) available to my jupyter installation in a different environment:
(pyth27) > python -m ipykernel install --prefix=/path/to/jupyter/env --name "python27"
Is there anything similar possible for the R kernel? So far I can only run the R kernel using a jupyter installation within the same environment(R_env).
One solution might be the nb-conda_kernels package. However there I'm not clear if it always adds all available kernels from all environments or whether I can specify which environments should be searched.
My question is similar to this one https://github.com/jupyter/jupyter/issues/397. Only that I don't want to use the base environment to start jupyter but a dedicated environment.
As described on https://github.com/IRkernel/IRkernel, the r-ikernel package provides a mechanism similar to python -m ipykernel install, to be run in R:
R> IRkernel::installspec()
To run this from Bash, you can do
(R_env)> Rscript -e "IRkernel::installspec()"
Now the tricky part, due to Jupyter and R being in different environments: According to https://github.com/IRkernel/IRkernel/issues/499, IRkernel::installspec() requires the jupyter-kernelspec command. I've tested two methods to provide it (to be done before issuing the above commands):
jupyter-kernelspec is part of Jupyter and hence in the file tree of jupyter_env, so add its path to PATH (I found it's better to add to the end so as to not disrupt other path lookups during the Rscript call)
(R_env)> export PATH="$PATH:</path/to/conda>/envs/jupyter_env/bin"
jupyter-kernelspec is included in the jupyter_client conda package, so you can do
(R_env)> conda install jupyter_client
Caveat: this installs a number of dependencies, including Python.
I opted for the first method to keep R_env free of Python packages.
I have installed the latest version of Anaconda on my Ubuntu Linux virtual machine.
I successfully launched Jupyter in my browser using the following command : ipython notebook .
But when I want to create a new notebook, I have to choose between Python [conda root] and Python [default].
So, what are the differences between them? Which one to choose to create a new Notebook?
(I want to use Jupyter to work with Apache Spark)
Thank you.
// Update 22/10/2016
I created a new Environment python2 using : conda create -n python2 python=2.7 anaconda and I activate this new one : source activate python2.
When I list all environments : $ conda info --envs, I see these two lines :
python2 * /home/opentuto/anaconda2/envs/python2
root /home/opentuto/anaconda2
Now, my notebook, look like this :
Any solution!!?
Python [conda root] and Python [default].
What you are seeing are notebook kernels that you can use to create notebooks. The kernels listed are dependent on anaconda environments that you create and kernels installed. You see only default and conda root, because I'm assuming that you have simply installed the newest version of Anaconda for python 2.7 or 3.5. When installing Anaconda, it creates a root virtual environment and you can change the default environment based on environments that you have set up.
See the following for more information related to your question:
How do I add python3 kernel to jupyter (IPython)
https://github.com/jupyter/notebook/issues/1716
Jupyter and Apache Spark
I recommend setting up jupyter and spark in the following manner w/ your .profile.
export SPARK_PATH=~/...
export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
After sourcing these variables you can call $ ./bin/pyspark and you'll have a spark context in a jupyter notebook.
The first approach is removing the extension... not recommanded.
The second approach is to just disable the extensions...recommanded
you can do something like this:
python -m nb_conda_kernels.install --disable --prefix=<ENV_PREFIX>
jupyter-nbextension disable nb_conda --py --sys-prefix
jupyter-serverextension disable nb_conda --py --sys-prefix
In this case, you just disable nb_conda_kernels and nb_conda that depend on the first one, but still keep the other ones active.
NOTE: Replace ENV_PREFIX with your root environment or another conda environment where the extensions have been installed.