I am using xarray with cfgrib to load grib files in Python. I have custom grib definitions, which I am providing to eccodes (backend for cfgrib) via the environment variable GRIB_DEFINITION_PATH.
This setup works well, as long as I run the Python script in an environment where the variable was already set.
Now I want to be more flexible with my setup and provide the environment variable from within Python using os.environ (see the example below).
But somehow when setting up the environment like this, the variable gets ignored and I don't understand why.
Can anyone provide me some insight into this mystery? Thanks in advance!
Here an "MRE" of the setting.
import xarray as xr
import os
grib_definitions_path = "/paths/to/definitions:/split/like/this"
os.environ["GRIB_DEFINITION_PATH"] = grib_definitions_path
grib_file = '/path/to/grib/file'
backend_args = {
"filter_by_keys": {"shortName": "P"}
}
array = xr.open_dataset(grib_file, engine="cfgrib", encode_cf=("geography", "vertical"), backend_kwargs=backend_args)["P"]
print(array.dims)
Executing the above code in a terminal fails for me with KeyError: 'P'. If I however first run
export GRIB_DEFINITION_PATH="/paths/to/definitions:/split/like/this"
the dimensions of array are being printed as expected.
I usually have to rerun (most parts of) a notebook when reopen it, in order to get access to previously defined variables and go on working.
However, sometimes I'd like to skip some of the cells, which have no influence to subsequent cells (e.g., they might comprise a branch of analysis that is finished) and could take very long time to run. These cells can be scattered throughout the notebook, so that something like "Run All Below" won't help much.
Is there a way to achieve this?
Ideally, those cells could be tagged with some special flags, so that they could be "Run" manually, but would be skipped when "Run All".
EDIT
%%cache (ipycache extension) as suggested by #Jakob solves the problem to some extent.
Actually, I don't even need to load any variables (which can be large but unnecessary for following cells) when re-run, only the stored output matters as analyzing results.
As a work-around, put %%cache folder/unique_identifier to the beginning of the cell. The code will be executed only once and no variables will be loaded when re-run unless you delete the unique_identifier file.
Unfortunately, all the output results are lost when re-run with %%cache...
EDIT II (Oct 14, 2013)
The master version of ipython+ipycache now pickles (and re-displays) the codecell output as well.
For rich display outputs including Latex, HTML(pandas DataFrame output), remember to use IPython's display() method, e.g., display(Latex(r'$\alpha_1$'))
Though this isn't exactly what you seem to be looking for, if you wish to entirely omit the execution of a cell (where no cached results are loaded), you can add the following hack at the beginning of a cell (assuming you are using a Unix-based OS):
%%script false
or a variant (working as of early 2020 -- see here for explanation):
%%script false --no-raise-error
Currently, there is no such feature included in the IPython notebook.
Nevertheless, there are some possibilities to make your life easier, like:
use the %store or maybe better the %%cache magic (extension) to store the results of these intermittently cells, so they don't have to be recomputed (see https://github.com/rossant/ipycache)
add a if==0: before the cells you don't want to execute
convert these cells to raw cells (but you will loose the already stored output!)
(see discussion at https://github.com/ipython/ipython/issues/2125)
Here's a simple and universal solution without the need for workarounds:
Simply type this as the top line of the cell to skip the cell:
%%script echo skipping
It's tested on Windows and Mac with recent Jupyter, and I think it should work on other Unix-like platforms as well because they also have an echo command. Some of the other proposed solutions are more platform-specific.
Of course you can put what ever text you like instead of "skipping". When you execute the cell, it will merely print this text instead of executing the code in the cell.
If no cached results are expected to be loaded, I find the Freeze nbextension quite useful to this end.
Although unofficial, I strongly recommend to give these notebook extensions a try if you have never used them before.
To install the extension machinery,
$ pip install jupyter_contrib_nbextensions && jupyter contrib nbextension install
To enable the Freeze extension, launch jupyter notebook and open a new notebook, from the menu select Edit > nbextensions config, and then check Freeze.
This question is a bit older, but the most convenient answer seems to be missing. You can use the 'initialization cells' from the NBextensions. Once installed/activated, you can in any notebook mark cells as 'initialization cells' which can then be run with a specific button.
Install NBextensions:here
Activate 'initialization cells' when you launch the jupyter dashboard
In your notebook, in the 'view' menu choose 'cell toolbar' then 'initialization cell'
Next to each cell there is now a checkbox. Mark all the cells you want to run for initialization
When reopening a notebook, click the button that looks like a pocket calculator to run all the initialization cells.
The simplest way to skip python code in jupyter notebook cell from running, I temporarily convert those cells to markdown.
The %%script false solution stopped working some time in 2019.
Here are some other available workarounds. These are based on programs ignoring their arguments when you tell them not to expect any. Here are some easy examples:
Perl:
%%perl -e0
for i in range(10): print(i)
Here you're running: perl -e '0' cellcontents
A more memorable version:
%%perl -eat
for i in range(10): print(i)
Here you're running: perl -e 'at' cellcontents
Bash:
%%bash -c :
for i in range(10): print(i)
: is a no-op in bash, so you're running: bash -c : cellcontents
I haven't looked at the external magic implementation code, but I'm pretty sure "cellcontents" are passed as arguments and won't be interpreted by shell by mistake, say if you were to include ; in them and accidentally inject some bad code. But I can't guarantee you that.
I'm sure you can come up with other creative solutions, by looking at the supported programs here: https://ipython.readthedocs.io/en/stable/interactive/magics.html#cell-magics
For the cells you wish to skip when you press Run All you can use try/except blocks, where you try to display the already calculated information in the try block, and perform the calculation in the except block.
Take this cell for example:
my_results = 2 # Assume this is a bigger calculation
print(my_results)
print('displayed my results.')
To skip the calculation for this cell in subsequent runs, change the contents of this cell to the following:
try:
print(my_results)
print('displayed state value')
except:
my_results = 2 # Assume this is a bigger calculation
print(my_results)
print('displayed newly calculated value')
The first time you run this cell, it will attempt to print the value of the state variable my_results. This throws an exception so it jumps to the except block and does the actual calculation of my_results (which in this case just makes it equate to 2). At the end of the first run the output for this cell would be:
2
displayed newly calculated value
When you run the cell a second time (whether that be manually or via Run All), the try block will first execute but this time since the variable is available in the state, it does not throw an exception. Instead it displays the result and the except block is never run. At the end of the second run the output of this cell would be:
2
displayed state value
This doesn't meet your explicit criteria that the cell should be completely skipped, but effectively the calculation is skipped.
If the displaying of the information is more complex than using a single print or display, it would probably be cleaner if you make the displaying routine into a function like so:
def print_my_results(my_result):
print(my_result)
print('displayed my result.')
try:
print_my_results(my_results)
except:
my_results = 2 # Assume this is a bigger calculation
print_my_results(my_results)
Coming over from a Matlab environment to VSCode. I really miss the ability to delete variables as needed. I think this is what's happening here. I'm trying to write an equation in vscode. One of the variables was misspelled as "simga"; I inteded it to be "sigma" (NOTE THE PLACEMENT OF THE "M"). For whatever reason, it seems like vscode keeps recognizing "simga". If this were matlab, I could delete all variables and start over. Not sure how to do this in vscode. Screenshot attached for reference.
Would appreciate suggestions,
Thanks!
You can write your own clearvars function.
test.py
import pandas as pd
A = 123
B = 224
def clearvars():
for el in sorted(globals()):
if '__' not in el:
print(f'deleted: {el}')
del el
clearvars()
python test.py
deleted: A
deleted: B
deleted: clearvars
deleted: pd
You can delete the print statement within the function if you want.
Used sources
dir inside function
Edit
If you have Jupyter installed and you can use an interactive window to see which variables are in the workspace.
You can create a jupyter code cell within your .py file adding #%% above the code you want to run. An interactive window will open with a tab variables.
If I testing my codes on IDLE the autocomplete its works but if I open a new file don't.
See there pictures below:
I just press CTRL + SPACE.
So.. don't work in this case:
I'll think there are some configuration for solve this, any one knows?
Python idle doesn't work that way. You get autocomplete in idle shell because values are deduced in every run. When you use files your program is not evaluated until you run. Because you can assign any type to a variable at run time, there is no way for idle to confirm the type of variable.
Understand with an example
>> a = dict()
>> a = set()
>> a. # <-- autocomplete knows type of a is set
but the same code in a file
a = dict()
a = set()
a. # <-- How does idle come to know what this variable is without running
but when you run this file once your global variables will show autocomplete feature, but not the local scope variables.
Have you tried saving the script as a *.py file before trying to use IDLE's autocomplete?
More than that, have you considered using a text editor with Python plugins, like Sublime Text and Atom? Or even a python-compatible IDE, like PyCharm, Spyder or even JupyterNotebook.
As my notebook gets longer, I want to extract some code out, so the notebook would be easier to read.
For example, this is a cell/function that I want to extract from the notebook
def R_square_of(MSE, kde_result):
# R square measure:
# https://en.wikipedia.org/wiki/Coefficient_of_determination
y_mean = np.mean(kde_result)
SS_tot = np.power(kde_result - y_mean,2)
SS_tot_avg = np.average(SS_tot)
SS_res_avg = MSE
R_square = 1 - SS_res_avg/SS_tot_avg
return R_square
How can I do it effectively?
My thoughts:
It's pretty easy to create a my_helper.py and put the code above there, then from my_helper import *
The problem is that, I may use other package in the method (in this case, np, i.e. numpy), then I need to re-import numpy in my_helper.py. Can it re-use the environment created in ipython notebook, hence no need for re-importing?
If I change the code in my_helper.py, I need to restart the kernel to load the change(NameError: global name 'numpy' is not defined), this makes it difficult to change code in that file.
Instead of importing your other file, you could instead run it with the %run magic command:
In [1]: %run -i my_helper.py
-i: run the file in IPython’s namespace instead of an empty one. This is useful if you are experimenting with code written in a text editor which depends on variables defined interactively.
I'd still take the opportunity to recommend writing the file as a proper python module and importing it. This way you actually develop a codebase usable outside of the notebook environment. You could write tests for it or publish it somewhere.