Develop Reference

Integrating Python and Jupyter Notebook with Visual Studio Code

Summary:
I am trying to set up the Jupyter Notebook extension for VSCode but I can’t get my Python code snippets to run. There is no "Run" button to select. I can't select a Python/ Jupyter kernel to work with.
The problem in greater detail:
If you check out this YouTube clip at exactly 1 minute and 46 seconds (here - - you can pause it to see the still) this is what I am trying to accomplish.
But here is what my VSCode actually looks like:
As you can see in my screenshot, the “Play” icon to the left of my code snippet is missing. Along the top border, the other “Play All” button is missing.
In Microsoft’s official VSC doc for setting up Jupyter Notebooks, right near the top under this specific heading: “Create or open a Jupyter Notebook” their interface has way more options at the top of the cells when compared to mine. They have buttons named: “Variables”, “Restart”, “Interrupt”. That is expected for Jupyter Notebook funcionality. Mine is missing all of that. Clearly there is something wrong with my setup.
Other SO users report having ms-python extension installed (which includes Jupyter functionality) in addition to stand-alone Jupyter Notebook extensions installed. Apparently in situations when you have multiple Python extensions installed, they can interfere with each other. So after much wrangling, I resolved to uninstalling all my Jupyter extensions and keeping only the main ms-python extension.
The issue persists.
Here is a complete list of all my VSC installed extensions:
$ code --list-extensions --show-versions
arcticicestudio.nord-visual-studio-code#0.19.0
batisteo.vscode-django#1.8.0
bibhasdn.django-html#1.3.0
Compulim.indent4to2#0.1.2
Equinusocio.vsc-community-material-theme#1.4.4
Equinusocio.vsc-material-theme#33.5.0
equinusocio.vsc-material-theme-icons#2.3.1
esbenp.prettier-vscode#9.8.0
formulahendry.code-runner#0.11.8
GulajavaMinistudio.mayukaithemevsc#3.2.3
icrawl.discord-vscode#5.8.0
janisdd.vscode-edit-csv#0.6.10
lonefy.vscode-JS-CSS-HTML-formatter#0.2.3
m-thorsen.vscode-materia#0.4.4
ms-python.python#2022.12.1
NikolaosGeorgiou.html-fmt-vscode#0.6.0
PKief.material-icon-theme#4.20.0
samuelcolvin.jinjahtml#0.17.0
streetsidesoftware.code-spell-checker#2.7.0
yzhang.markdown-all-in-one#3.5.0
zamerick.black-ocean#1.0.2
zhuangtongfa.material-theme#3.15.2
There is a bit of noise in that list. I'm not sure what all of them refer to. But for certain there is only one Python app (ms-python) and nothing related to Jupyter.
Some SO members elsewhere have suggested selecting / setting the base Python version/environment. When I click the “Select Kernel” button at the top right (as illustrated in the screenshot below), it reveals an empty menu (see at the top left):
Here is “Python v3.8.5 (conda)” showing as selected at the bottom and slightly to the left:
Another exhaustive question and answer thread elsewhere on Stack Overflow involving an issue similar to mine includes many, many creative potential solutions: jupyter server : not started, no kernel in vs code
In that Stack Overflow question, one prominent member answers:
How I solved it:
Press Command+Shift+P to open a new command pallete
Type >Python: Select Intepreter to start jupyter notebook server
Open the notebook again
Someone else suggests:
Press Command+Shift+P on mac, Ctrl+Shift+p on windows
Type Jupyter: Select Interpreter to start Jupyterserver
It would show you a dropdown of python versions installed.
I chose python 3.7.5 and it worked for me. you can choose the python version installed on your machine.
Another member says:
I have seen all possible solutions but not work, finally I just
upgrade jupyter, notebook, and jupyterlab, like pip install -U jupyterlab, and I can choose the kernel in VScode!
None of the above resolves my issue.
Here is a quirky new twist: After installing jupyterlab with pip, after restarting VSC, I encountered this weird error:
It’s a condition box. I clicked: “Yes”.
Jupyter is still not integrated properly.
UPDATED: Based on JialeDu’s answer, I followed his instructions. I created a video shared below. It doesn’t quite capture every step, but I did follow all the steps. What you can see in the gif video is that my current working directory is a folder called Test, a new terminal has been opened, and there is an active virtual environment with $ source .venv/bin/activate. Next, as you can see in the .gif, I am able to select the latest Python interpreter v3.10.5 showing as “Recommended”. So that is progress. I couldn’t do that before. Going the virtual environment route was a great idea. However, I am still not able to get the Jupyter Notebook interface as the end product shared by JialeDu. When I invoke Ctrl + Shift + P again and type > create jup a new Jupyter Notebook file is created but the kernel I selected previously does not show and the expected Jupyter interface buttons and options are still not showing. Here is the video demo in .gif formatting:
Please note: Here you can see a list of 4 available Jupyter extensions:
I avoided installing any of these separate Jupyter extensions in the store because according to the instructor (and as confirmed on the official Microsoft Python extension ms-python page which I already have), it includes Jupyter built in. From the ms-python extension entry, it explains: "This Python extension will automatically install the Pylance and Jupyter extensions to give you the best experience when working with Python files and Jupyter notebooks."
Even though other Stack Overflow users indicated that mixing the above extensions such as ms-python with ms-toolsai was a bad idea, I experimented with having them both installed anyway. Didn’t fix the issue. I tried uninstalling ms-python and keeping ms-toolsai. Still no dice.
By the way, I’ve also got the latest version (as of this writing) of VSCode v1.70.1 installed:

VS Code has prompted you to install the suggested extension. Python and Jupyter.
Also it helps to make sure your vscode is up to date.
The play button of the interface is provided by Jupyter, but if you want to run the code (select the kernel), you must also install the Python extension.
It’s a condition box. I clicked: “Yes”.
You don't have to click Yes because there is a time delay. By the time you click Yes, the environment may not be ready.
If you are using a virtual environment, you can select the virtual environment's interpreter and then create a new terminal to activate the environment.
Complete creation steps:
Prerequisites:
The machine has python installed,
and vscode has Python and Jupyter extensions installed.
steps
Create a new working folder Test
Open the folder Test with vscode
new terminal
Type the command python -m venv .venv to create a virtual environment
After the creation is complete, use the command .venv/scripts/activate to activate the virtual environment
Or Ctrl+Shift+P --> Python:Select Interpreter, select the interpreter of the virtual environment and then create a new terminal to activate the environment
Create new jupyter file using command palette command Create:New Jupyter Notebook
Or right click --> New File --> name ends with .ipynb
The virtual environment you just selected will be enabled as the kernel by default.

I discovered a very simple answer: I installed the VSC package in the official Snap repo.
VSC and Python-Jupyter look to be all playing nice together now:

Can’t create an environment (Conda or pipenv) that works properly with Jupyter

The long-short:
One day in July, noticed that Jupyter wasn’t importing the version of Seaborn I installed to my Conda env. It was downloading an older Seaborn from a global dir. Same for all other packages when I checked versions. After various attempts at fixing this, Jupyter doesn’t even import packages now. I’ve tried with pipenv too. In both Conda and pipenv, sys.path reveals more path variables than I know what to do with, sometimes including the desired env path, sometimes not. But either way Jupyter imports are ignoring the env path I want to use, instead looking for global packages that I have deleted since July in attempt to solve the issue. On top of this (but probably intertwined in a way I just don’t understand yet), I’m not getting kernels that connect Jupyter to the desired env directories where packages are installed. Global python gets used instead of the env’s python instance. Not sure exactly how kernels are created, but I can tell that they are either not being created for some new envs, not accessing env-specific python & packages, or failing to connect (loads a stale kernel.json file and fails to start.)
Desired outcome:
How do I get JupyterLab to import the intended package version from the intended env directories? Even deeper, How do I get environments back to their former functionality of automatically [1] creating their own python instance, [2] creating their own kernel recognized by Jupyter, [3] creating their own path to the env, and [4] initializing all of that in Jupyter Lab?
Things I have tried:
Deactivated the base env, which admittedly I wasn’t doing for the first few weeks of July until I remembered that is a must for Windows Conda… but still now that I’m regularly deactivating base, why would these issues still persist?
Uninstalled Anaconda and reinstalled Miniconda
Deleted (I believe) all stray / older pythons from my machine, reinstalled a fresh user-level Python 3.9
Verified that packages are installing with the conda list command. They just won’t import properly in JupyterLab
Some tweaks to Path variables in Windows settings, but was very cautious and have no idea how to / if I am revising those properly. Clearly not though seeing as the issue is still alive 3 months later 0.o No idea if I should be editing user variables or system variables, or how to trim the paths cautiously.
Reinstalled jupyter and jupyterlab
Reinstalled ipykernel
Noted that sys.executable, sys.path, and !where python give different outputs in shell python versus Jupyter Lab python
Switching kernels manually in JupyterLab (usually the manually selected kernel is DOA)
Tried manually rewriting sys.path in notebook.
Tried launching JupyterLab from a fresh pipenv instead of Miniconda
Got a pipenv working correctly for a Streamlit applet in August, using raw python in VS Code (This may isolate the issue to Jupyter? I had to continue the other Jupyter-based project in Google collab since JupyterLab started choking in July.)
Deeper details:
Sheesh this post is already getting long… but here I’ve picked out the 4 most prominent code / error snippets. There are more where that came from, but hopefully there might be something in here that you might recognize as the needle in the haystack.
[1]
When the kernel-session.json causes a stillborn kernel, it is created as a totally blank json file. Running jupyter lab in the conda terminal yields this error message, buried in the output:
Failed to load connection file:
'C:\\Users\\David.000\\AppData\\Roaming\\jupyter\\runtime\\kernel-a6082e80-0b65-48f1-b370-7c2918030185.json'
I’ve found past kernel-session.json files that were not stillborn, such as this one called kernel-b146b600-81e3-418e-a55f-5a3fbbc13471.json, which looks like this, auto-populated:
{
"shell_port": 50877,
"iopub_port": 50878,
"stdin_port": 50879,
"control_port": 50880,
"hb_port": 50881,
"ip": "127.0.0.1",
"key": "caa915a4-00a599e8b6c4db6417bcca77",
"transport": "tcp",
"signature_scheme": "hmac-sha256",
"kernel_name": ""
}
[2]
sys.executable, when run in shell python, yields the proper python location within the env:
>>> sys.executable
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\python.exe'
But sys.executable, when run in Jupyter Lab, is latching onto the global Python:
sys.executable
'C:\\Python39\\python.exe'
[3]
sys.path, when run in shell python, seems to yield the env specific paths I want (though why are there so many??)
>>> sys.path
['', 'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\python39.zip',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\DLLs',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\lib',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml',
'C:\\Users\\David.000\\AppData\\Roaming\\Python\\Python39\\site-packages',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\lib\\site-packages',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\lib\\site-packages\\win32',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\lib\\site-packages\\win32\\lib',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\lib\\site-packages\\Pythonwin']
But sys.path, when run in Jupyter Lab, is also showing a butt-load of paths, many of which are unwanted globals:
sys.path
[> ['C:\\Users\\David.000\\Desktop\\Civic Innovation Corps\\Miami\\Predictive_Analytics_Business_Licensing',
'C:\\Python39\\python39.zip',
'C:\\Python39\\DLLs',
'C:\\Python39\\lib',
'C:\\Python39',
'',
'C:\\Users\\David.000\\AppData\\Roaming\\Python\\Python39\\site-packages',
'C:\\Python39\\lib\\site-packages',
'C:\\Python39\\lib\\site-packages\\win32',
'C:\\Python39\\lib\\site-packages\\win32\\lib',
'C:\\Python39\\lib\\site-packages\\Pythonwin',
'C:\\Python39\\lib\\site-packages\\IPython\\extensions',
'C:\\Users\\David.000\\.ipython']

I'm probably not qualified to answer this but what the heck.
I have several kernels associated with different environments that are configured to run different programs. I created them in the terminal and I make it a practice to install any module/program etc, within the terminal in the activated environment that I need it in instead of installing within the notebook.
If I'm using a desktop, then before opening Juypter I select the appropriate environment within anaconda. If I'm using a virtual machine then within the terminal I activate the environment and once I'm in the notebook, select the appropriate kernel.
While I can't be sure, I think your problem is occurring because of how/where you're installing or updating packages.

Conda environment kernels disappearing when called

EDIT: Problem solved.
Turns out Avast was classifying python.exe within the environments as a virus due to fresh conda installs (and being in silent mode I never found out). Solution found from here: https://github.com/ContinuumIO/anaconda-issues/issues/8266
__
I have something of a problem which I haven't seen anyone else directly have with conda environments and python kernels.
I can successfully create environments and install packages in them etc., but everytime I try to either (i) start a jupyter notebook or (ii) otherwise call an environment kernel in e.g. the VS Code jupyter extension, the kernel environment kernel "disappears" entirely from the environment in question.
In the former jupyter case, the environment successfully launches the notebook browser window when calling "jupyter-notebook" for the first time, but after some 5-10 seconds the browser dies and gives a connection error (I allowed my browser through firewalls etc. which hasn't helped). Trying to call the notebook again from the command line results in an "Unable to create process using 'C:\Users\user\miniconda3\envs\envname\python.exe C:\Users\user\miniconda3\envs\envname\Scripts\jupyter-notebook-script.py' error, after which calling "where python" on the command line only returns the base kernel, and loses all connection to the environment kernel.
In the VS Code case using the jupyter extension, I correctly get prompted to choose a python interpreter and successfully manage again to execute cells for about 5-10 seconds, until VS Code returns an "Error: Direct kernel connection broken" after which again the kernel just completely vanishes from both the command line and the kernel menu in VS Code.
No files in the environments seem to have changed (AFAIK) and I am completely baffled as to what is going on.
Throughout all of these the base kernel seems to run fine, so I'd venture a guess this has something to do with the way jupyter calls the environment kernel and somehow breaks them in the process. I was successfully running a number of environments for months before on the same setup. The problem initially occurred when I created a new environment, launched VS Code for that environment which inadvertently did not have ipykernel installed, and after getting a prompt from VS Code asking me to install ipykernel, every pre-existing environment consistently started failing in the way described above. After this I reinstalled Anaconda and I have been unable to get a single environment to work.
A minimal example of how the environments were typically created:
conda create --name envname_1 python=3.7
and in testing and installing some packages with conda I cloned the base environment a number of times to sequentially test whether the kernel keeps failing with
conda create --name envname_2 --clone envname_1
and so on. Regardless of the environment the problem persists. After activating the environment with
conda activate envname
asking "where python" within the environment initially correctly returns
C:\Users\user\miniconda3\envs\envname\python.exe
C:\Users\user\miniconda3\python.exe
and after calling the kernel in some jupyter-context and having it fail, "where python" from the command line returns just
C:\Users\user\miniconda3\python.exe
I am running on Windows 10 Home 64-bit (version 2004, OS build 19041.867), miniconda3 4.9.2 with base environment python 3.8.5 (the environment kernels tested have mainly been 3.7.x, but 3.8.x has not made a difference so far in testing). Miniconda is installed in my C:\Users\user\miniconda3 directory. (I'm not sure what other details would be helpful here).
So far I've tried:
Purging everything conda-related or python-related from my computer as completely as possible, by using 1) anaconda-clean and then removing the "envs" and "pkgs" folders separately before running the uninstall from standard conda. Miniconda is just the latest n-th reinstall after which the same issues persist.
Setting up a number of test environments in different ways and sequentially testing where the error occurs - seems to always be when activating the kernel one way or another
Removing all possibly related path variables prior to reinstalling
Removing all other python versions (which correctly started pointing python to the base environment - before this an additional problem was the each environment on the command line was attempting to point to my global python installation instead of either the base or environment kernels - this might be somehow related to what is going on here)
Trying all kinds of other little things related to jupyter-notebooks losing kernels within the notebooks, but mostly those have not been applicable since I cannot even use the command line after calling the kernel to work once.
Verified at each step when creating new environments that the environment kernel is correctly pointed to in the environment until calling it from jupyter or VS Code (+ jupyter lab seems to have the same outcome)
I'm a humble man and don't ask for much, would just like to keep working with isolated environments in the future as well. I'm not exactly a whiz when it comes to these things so I'm unsure where else I should be looking or what else I should be trying. Am starting to lose hope as I've been trying to figure this out for a good week now with nothing to show for it.
I hope I'm just being an idiot and overlooking something simple and obvious here since I've not managed to find any similar issues having been resolved. Regardless, any ideas on where I might look to next would be invaluable!

How to fix Visual Studio Code Jupyter notebook error?

VScode for python runs perfectly, without any error, problems starts whenever I try to run Jupyter notebook inside VScode.
It starts with Ipython Kernel loading and then it shows Command Failed Error. Any fix for this ? I have no idea what's the problem and I don't know much about the environment and stuffs also.
Thank You.

From your picture, VSCode said Command failed, and it's a very long command. It mainly includes two parts: activate the conda base environment and change the vscode python environment through "Python" extension. And from the top right of the picture, it shows the jupyter server start failed.
So you can try to do two things:
First one: change the environment. Instead of using conda, download and install a global python from the official website. And recommend creating a virtual environment. Then select the environment instead of conda base environment.
Second one: reinstall or downgrade the 'Python' extension(if the first one not work).
This problem can be caused by the package in the environment, so I recommend you just change to another environment.

how to use spark with python or jupyter notebook

I am trying to work with 12GB of data in python for which I desperately need to use Spark , but I guess I'm too stupid to use command line by myself or by using internet and that is why I guess I have to turn to SO ,
So by far I have downloaded the spark and unzipped the tar file or whatever that is ( sorry for the language but I am feeling stupid and out ) but now I can see nowhere to go. I have seen the instruction on spark website documentation and it says :
Spark also provides a Python API. To run Spark interactively in a Python interpreter, use bin/pyspark but where to do this ? please please help .
Edit : I am using windows 10
Note:: I have always faced problems when trying to install something mainly because I can't seem to understand Command prompt

If you are more familiar with jupyter notebook, you can install Apache Toree which integrates pyspark,scala,sql and SparkR kernels with Spark.
for installing toree
pip install toree
jupyter toree install --spark_home=path/to/your/spark_directory --interpreters=PySpark
if you want to install other kernels you can use
jupyter toree install --interpreters=SparkR,SQl,Scala
Now run
jupyter notebook
In the UI while selecting new notebook, you should see following kernels availble
Apache Toree-Pyspark
Apache Toree-SparkR
Apache Toree-SQL
Apache Toree-Scala

When you unzip the file, a directory is created.
Open a terminal.
Navigate to that directory with cd.
Do an ls. You will see its contents. bin must be placed
somewhere.
Execute bin/pyspark or maybe ./bin/pyspark.
Of course, in practice it's not that simple, you may need to set some paths, like said in TutorialsPoint, but there are plenty of such links out there.

I understand that you have already installed Spark in the windows 10.
You will need to have winutils.exe available as well. If you haven't already done so, download the file from http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe and install at say, C:\winutils\bin
Set up environment variables
HADOOP_HOME=C:\winutils
SPARK_HOME=C:\spark or wherever.
PYSPARK_DRIVER_PYTHON=ipython or jupyter notebook
PYSPARK_DRIVER_PYTHON_OPTS=notebook
Now navigate to the C:\Spark directory in a command prompt and type "pyspark"
Jupyter notebook will launch in a browser.
Create a spark context and run a count command as shown.