Ok, this must be silly but I really can't find a good way to do it.
When I open a Python session with Jupyter Notebook or a Jupyter Lab, I cannot have access to all my scripts on other folders which are generally included in PYTHONPATH.
If I run the Jupyter Notebook from Pycharm, it actually works and I can see that I have the correct PYTHONPATH by doing
import os
os.getcwd()
but if I do the same with a session started from CLI, I get a KeyError and PYTHONPATH not found.
This must have been solved a thousand times, how come I can't find anything about it?
Related
I'm trying to run some code on VSCode, as I'm shifting over from PyCharm and Vim to utilise VSCode as a general purpose editor for a variety of programming languages.
I'm currently trying to make some code work, which should have been quite straight forward given it is a project in my PyCharm IDE and I have been able to run it there.
When I open up my VScode workspace, and try and run the code, it cannot find a directory that obviouslly exists:
So I'm getting this error:
Exception has occurred: FileNotFoundError
data/connectivity/weights.txt not found.
And, my code is also pretty straight forward, its something like this:
SC = np.loadtxt(join("data/connectivity", "weights.txt"))
numpy is loaded in as np. and os.path.join is loaded in as join.
The directory data/ is in the same parent directory as the code/script I'm running.
I have no idea why it isn't recognising the path, or the folder.
**NOTE: **
To add, when I run my code in jupyter notebook in VSCode, it runs smoothly and recognises the path.
The jupyter notebook is also in the same parent directory as the data/ directory. So there is no visible difference between the .py and .ipynb files regarding code and relative position to the data/ directory.
Any help would be much appreciated. There is probably a simple solution that I'm failing to catch onto at the moment.
Thanks,
Boki
I've tried changing the directory to an absolute path on my Mac, which didn't work.
I also tried--though probably not so usefully--adding __init__py files to each of the subdirectories, but as they are not modules, this was sort of pointless, I was just trying to be exhaustive.
As I mention above, code works perfectly when I run the same code as a .ipynb, or when I run it in my old IDE: PyCharm
This is because the current directory of the terminal is different when vscode and pycharm run files.
By default, vscode executes files in the workspace, and pycharm executes files in the folder where the files are run.
You can search Terminal: Execute In File Dir in settings and check it.
This will be consistent with pycharm.
Of course, the default setting of vscode is also beneficial. When encountering multi-level directories, it is more convenient to import files.
The long-short:
One day in July, noticed that Jupyter wasn’t importing the version of Seaborn I installed to my Conda env. It was downloading an older Seaborn from a global dir. Same for all other packages when I checked versions. After various attempts at fixing this, Jupyter doesn’t even import packages now. I’ve tried with pipenv too. In both Conda and pipenv, sys.path reveals more path variables than I know what to do with, sometimes including the desired env path, sometimes not. But either way Jupyter imports are ignoring the env path I want to use, instead looking for global packages that I have deleted since July in attempt to solve the issue. On top of this (but probably intertwined in a way I just don’t understand yet), I’m not getting kernels that connect Jupyter to the desired env directories where packages are installed. Global python gets used instead of the env’s python instance. Not sure exactly how kernels are created, but I can tell that they are either not being created for some new envs, not accessing env-specific python & packages, or failing to connect (loads a stale kernel.json file and fails to start.)
Desired outcome:
How do I get JupyterLab to import the intended package version from the intended env directories? Even deeper, How do I get environments back to their former functionality of automatically [1] creating their own python instance, [2] creating their own kernel recognized by Jupyter, [3] creating their own path to the env, and [4] initializing all of that in Jupyter Lab?
Things I have tried:
Deactivated the base env, which admittedly I wasn’t doing for the first few weeks of July until I remembered that is a must for Windows Conda… but still now that I’m regularly deactivating base, why would these issues still persist?
Uninstalled Anaconda and reinstalled Miniconda
Deleted (I believe) all stray / older pythons from my machine, reinstalled a fresh user-level Python 3.9
Verified that packages are installing with the conda list command. They just won’t import properly in JupyterLab
Some tweaks to Path variables in Windows settings, but was very cautious and have no idea how to / if I am revising those properly. Clearly not though seeing as the issue is still alive 3 months later 0.o No idea if I should be editing user variables or system variables, or how to trim the paths cautiously.
Reinstalled jupyter and jupyterlab
Reinstalled ipykernel
Noted that sys.executable, sys.path, and !where python give different outputs in shell python versus Jupyter Lab python
Switching kernels manually in JupyterLab (usually the manually selected kernel is DOA)
Tried manually rewriting sys.path in notebook.
Tried launching JupyterLab from a fresh pipenv instead of Miniconda
Got a pipenv working correctly for a Streamlit applet in August, using raw python in VS Code (This may isolate the issue to Jupyter? I had to continue the other Jupyter-based project in Google collab since JupyterLab started choking in July.)
Deeper details:
Sheesh this post is already getting long… but here I’ve picked out the 4 most prominent code / error snippets. There are more where that came from, but hopefully there might be something in here that you might recognize as the needle in the haystack.
[1]
When the kernel-session.json causes a stillborn kernel, it is created as a totally blank json file. Running jupyter lab in the conda terminal yields this error message, buried in the output:
Failed to load connection file:
'C:\\Users\\David.000\\AppData\\Roaming\\jupyter\\runtime\\kernel-a6082e80-0b65-48f1-b370-7c2918030185.json'
I’ve found past kernel-session.json files that were not stillborn, such as this one called kernel-b146b600-81e3-418e-a55f-5a3fbbc13471.json, which looks like this, auto-populated:
{
"shell_port": 50877,
"iopub_port": 50878,
"stdin_port": 50879,
"control_port": 50880,
"hb_port": 50881,
"ip": "127.0.0.1",
"key": "caa915a4-00a599e8b6c4db6417bcca77",
"transport": "tcp",
"signature_scheme": "hmac-sha256",
"kernel_name": ""
}
[2]
sys.executable, when run in shell python, yields the proper python location within the env:
>>> sys.executable
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\python.exe'
But sys.executable, when run in Jupyter Lab, is latching onto the global Python:
sys.executable
'C:\\Python39\\python.exe'
[3]
sys.path, when run in shell python, seems to yield the env specific paths I want (though why are there so many??)
>>> sys.path
['', 'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\python39.zip',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\DLLs',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\lib',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml',
'C:\\Users\\David.000\\AppData\\Roaming\\Python\\Python39\\site-packages',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\lib\\site-packages',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\lib\\site-packages\\win32',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\lib\\site-packages\\win32\\lib',
'C:\\Users\\David.000\\miniconda3\\envs\\aucu_ml\\lib\\site-packages\\Pythonwin']
But sys.path, when run in Jupyter Lab, is also showing a butt-load of paths, many of which are unwanted globals:
sys.path
[> ['C:\\Users\\David.000\\Desktop\\Civic Innovation Corps\\Miami\\Predictive_Analytics_Business_Licensing',
'C:\\Python39\\python39.zip',
'C:\\Python39\\DLLs',
'C:\\Python39\\lib',
'C:\\Python39',
'',
'C:\\Users\\David.000\\AppData\\Roaming\\Python\\Python39\\site-packages',
'C:\\Python39\\lib\\site-packages',
'C:\\Python39\\lib\\site-packages\\win32',
'C:\\Python39\\lib\\site-packages\\win32\\lib',
'C:\\Python39\\lib\\site-packages\\Pythonwin',
'C:\\Python39\\lib\\site-packages\\IPython\\extensions',
'C:\\Users\\David.000\\.ipython']
I'm probably not qualified to answer this but what the heck.
I have several kernels associated with different environments that are configured to run different programs. I created them in the terminal and I make it a practice to install any module/program etc, within the terminal in the activated environment that I need it in instead of installing within the notebook.
If I'm using a desktop, then before opening Juypter I select the appropriate environment within anaconda. If I'm using a virtual machine then within the terminal I activate the environment and once I'm in the notebook, select the appropriate kernel.
While I can't be sure, I think your problem is occurring because of how/where you're installing or updating packages.
New to python, so bear with me.
I have installed Anaconda and Jupyter Notebook successfully, and I have also set up what I think is the correct Path (at least for Anaconda). From an Anaconda prompt, python works as expected, and I can import the modules I need. However, I get a ModuleNotFoundError when I try to import certain modules in Jupyter Notebook (such as cx_Oracle). I think the issue is they are running off of two different directories.
When I run sys.executable in the Anaconda prompt, I get this directory: C:\Users\username\Documents\Anaconda\python.exe
When I run sys.executable in Jupyter Notebook, I get this directory: C:\Users\username\AppData\Local\Continuum\anaconda3\python.exe
Shouldn't these be the same? I would like Jupyter Notebook to execute on the same directory the Anaconda prompt does: C:\Users\username\Documents\Anaconda\python.exe
So my question is: how do I change the Jupyter Notebook directory so it points to the same one Anaconda is using?
Thanks all. This issue has been driving me crazy.
I'm new to using Anaconda and Spyder and this must be a simple setting issue, but it's really frustrating. I need to have a PythonPath environment set because I frequently run scripts in ArcGIS which won't work without a path. However, neither Anaconda Navigator nor Spyder will not open if there is a PythonPath set. I get the splash screen, but that's it, it doesn't open. If I remove the PythonPath and reboot, Anaconda and Spyder work, but not ArcGIS. This is not an ArcGIS problem, rather Anaconda and Spyder just don't seem to like there being a PythonPath at all. Anyone have experience with this?
For reference, my PythonPath contains:
C:\Python27;
C:\Python27\Lib;
C:\Python27\Lib\site-packages;
C:\Python27\DLLs;
C:\Python27\Lib\lib-tk;
C:\Python27\scripts;
C:\Python27\ArcGIS10.3\Lib;
C:\Python27\ArcGIS10.3\Lib\lib-tk;
C:\Python27\ArcGIS10.3\Lib\site-packages;
C:\Program Files (x86)\ArcGIS\Desktop10.3\arcpy;
C:\Program Files (x86)\ArcGIS\Desktop10.3\bin
I can remove everything except the bottom five and ArcGIS scripts still work, but it doesn't fix the problem with Spyder. It probably has something to do with the files not actually being executables, but links to scripts. For example, the shortcut for Spyder points to:
C:\ProgramData\Anaconda2\pythonw.exe C:\ProgramData\Anaconda2\cwp.py
C:\ProgramData\Anaconda2 "C:/ProgramData/Anaconda2/pythonw.exe"
"C:/ProgramData/Anaconda2/Scripts/spyder-script.py"
which seems to leave a lot of room for trouble. Interestingly, if I run from the Command Prompt, the first time there are no errors but nothing happens, and the second time it displays:
Spyder is already running. If you want to open a new instance, please
pass to it the --new-instance option
I don't know that I can directly offer a fix, but a nice workaround is spyder's PYTHONPATH manager inside the program. If you open up spyder you should be able to locate the pythonpath manager (on a mac it is under "python" tab) and manually add the necessary paths.
You seem to have relic python installation kicking around and it wasn't uninstalled prior to installing ArcMap. The correct path structure for a clean installation of ArcMap with its associated python would consist of
C:\Python27\ArcGISxx.x
and there are no other files or folders in the C:\Python27 path.
I would suggest uninstalling python completely and do a 'repair' of the arcmap distribution which will reinstall python and its dependencies correctly.
In the interim, you should move the last 3 lines (which refer to arcmap) to the top of the dependency list. That may resolve things temporarily.
I have a custom Jupyter kernel which runs IPython using a custom IPython profile which uses a matplotlib stylesheet.
I know to run this successfully normally I would put:
The matplotlib stylesheet in ~/.config/matplotlib/stylelib/
The IPython profile in ~/.ipython/
The kernel json in ~/.jupyter/kernels/my_kernel/
But I am doing this as part of larger program which runs in a virtualenv, and if I put the things as above then any notebook server running on the computer will be able to see the custom kernels, even if it is running outside the venv. I don't what this because I don't want my program to interfere with other notebooks on the computer.
I think what I need to do is put the things above somewhere equivalent inside the venv but I can't figure out where they should go. Doe anyone know where they would go? Or is this just a thing IPython/Jupiter can't/won't do?
It's probably worth mentioning that in the case of the stylesheet for example I don't want to just put it in the working directory of my program (which is one option matplotlib offers).
You can put kernelspecs in VIRTUAL_ENV/share/jupyter/kernels/ and they will be made available if the notebook server is running in that env. In general, <sys.prefix>/share/jupyter/kernels is included in the path to look for kernelspecs.
You can see the various locations Jupyter will look, you can see the output of jupyter --paths:
$ jupyter --paths
config:
/Users/you/.jupyter
/Users/you/env/etc/jupyter
/usr/local/etc/jupyter
/etc/jupyter
data:
/Users/you/Library/Jupyter
/Users/you/env/share/jupyter
/usr/local/share/jupyter
/usr/share/jupyter
runtime:
/Users/you/Library/Jupyter/runtime
Kernelspecs are considered data files, and will be found in any of those directories listed under data:, in a kernels subdirectory, e.g. /usr/local/share/jupyter/kernels.