Develop Reference

Turn a fits file into healpix map

I created a fits file. It is just a 2D array, I can visualize it using plt.imshow(fits.getdata(my_file)). Is there a way to turn it into a healpix map? If yes, please provide a detailed answer. If no, please explain why. Any help appreciated!
I am aware of healpy.fitsfunc.write_map(filename, m) but I struggle with using it (cannot set the m parameter) and do not know if this function is any help with my task

The hp.write_map function is only to write a pre-existing healpix map from memory into a fits file.
From your question, It is not clear what sort of data you have. But assuming that you have a 2d grid of data since you use imshow. You need to convert it to a healpix map before you use hp.write_map.
Depending on your coordinate system, you need to know the coordinates of each of the grid points, to convert the grid into healpix map.
To convert some data (scalar) into a healpix map, after finding the coordinates, You can write a function like this, which takes the coordinates and makes a healpix map for you.
def make_map_vec(theta, phi, data):
assert len(theta) == len(phi) == len(data)
e1map = np.full(hp.nside2npix(NSIDE), hp.UNSEEN, dtype=np.float)
index = hp.ang2pix(NSIDE, theta, phi)
values = np.fromiter((np.sum(data[index==i]) for i in np.unique(index)), float, count=len(np.unique(index)))
e1map[np.unique(index)] = values
return e1map
This link gives you the coordinate system used by healpix.

I have the same problem, I want to plot a 2d array in mollweide projection.
My 2d array is a 180*360 array, using matplotlib.pyplot. imshow the axis don't follow the projection:
so to resolve this problem I want to use mollview of healpy but I don't know to convert my 2d array Data(180*360) into an object that mollview could plot.
In the previous answer, you assume data is 1D array having same length as theta and phi. I tried to test it doing this :
test = np.ones((180,360))
data = test.reshape(180*360)
theta = []
phi = []
for i in range (360):
for j in range (180):
theta.append((j)*np.pi/180)
phi.append((i)*np.pi/180)
theta = np.asarray(theta)
phphi = np.asarray(y)
def make_map_vec(theta, phi, data):
assert len(theta) == len(phi) == len(data)
e1map = np.full(hp.nside2npix(NSIDE), hp.UNSEEN, dtype=np.float)
index = hp.ang2pix(NSIDE, theta, phi)
values = np.fromiter((np.sum(data[index==i]) for i in np.unique(index)), float, count=len(np.unique(index)))
e1map[np.unique(index)] = values
return e1map
map_test = make_map_vec(theta,phi,data)
hp.mollview(map_test,title="Mollview image RING")
I got :
which is not correct.
Do you know if there is any other way to get a map from a 2D array?

Python/SciPy: Issues converting DataFrame from polar to Cartesian grid

I have measurements (PPI arc scans) taken with a doppler wind lidar. The data is stored in a pandas dataframe where rows represent azimuth angle and columns represent radial distance (input shape = 30x197). Link to example scan, (csv). I want to transform this to a cartesian coordinate system, and output a 2d array which is re-gridded into x,y coordinates instead of polar with the values stored in the appropriate grid cell. Interpolation (nearest neighbor) is ok and so is zero or NaN padding of areas where no data exists.
Ideally the X and Y grid should correspond to the actual distances between points, but right now I'm just trying to get this working. This shouldn’t be terribly difficult, but I’m having trouble obtaining the result I want.
So far, I have working code which plots on a polar axis beautifully (example image) but this won't work for the next steps of my analysis.
I have tried many different approaches with scipy.interpolate.griddata, scipy.ndimage.geometric_transform, and scipy.ndimage.map_coordinates but haven't gotten the correct output. Here is an example of my recent attempt (df_polar is the csv file linked):
# Generate polar and cartesian meshgrids
r = df_polar.columns
theta = df_polar.index
theta = np.deg2rad(theta)
# Polar meshgrid
rad_c, theta_c = np.meshgrid(r,theta)
# Cartesian meshgrid
X = rad_c * np.cos(theta_c)
Y = rad_c * np.sin(theta_c)
x,y = np.meshgrid(X,Y)
# Interpolate from polar to cartesian grid
new_grid = scipy.interpolate.griddata(
(rad_c.flatten(), theta_c.flatten()),
np.array(df_polar).flatten(), (x,y), method='nearest')
The result is not correct at all, and from reading the documentation and examples I don't understand why. I would greatly appreciate any tips on where I have gone wrong. Thanks a lot!!

I think you might be feeding griddata the wrong points. It wants cartesian points and if you want the values interpolated over a regular x/y grid you need to create one and provide that too.
Try this and let me know if it produces the expected result. It's hard for me to tell if this is what it should produce:
from scipy.interpolate import griddata
import pandas as pd
import numpy as np
df_polar = pd.read_csv('onescan.txt', index_col=0)
# Generate polar and cartesian meshgrids
r = pd.to_numeric(df_polar.columns)
theta = np.deg2rad(df_polar.index)
# Polar meshgrid
rad_c, theta_c = np.meshgrid(r, theta)
# Cartesian equivalents of polar co-ordinates
X = rad_c*np.cos(theta_c)
Y = rad_c*np.sin(theta_c)
# Cartesian (x/y) meshgrid
grid_spacing = 100.0 # You can change this
nx = (X.max() - X.min())/grid_spacing
ny = (Y.max() - Y.min())/grid_spacing
x = np.arange(X.min(), X.max() + grid_spacing, grid_spacing)
y = np.arange(Y.min(), Y.max() + grid_spacing, grid_spacing)
grid_x, grid_y = np.meshgrid(x, y)
# Interpolate from polar to cartesian grid
new_grid = griddata(
(X.flatten(), Y.flatten()),
df_polar.values.flatten(),
(grid_x, grid_y),
method='nearest'
)
The resulting values look something like this (with grid_spacing = 10 and flipping x and y):
import matplotlib.pyplot as plt
plt.imshow(new_grid.T, cmap='hot')
Clearly interpolate "nearest" needs taming...

Finding the closest ground pixel on an irregular grid for given coordinates

I work with satellite data organised on an irregular two-dimensional grid whose dimensions are scanline (along track dimension) and ground pixel (across track dimension). Latitude and longitude information for each ground pixel are stored in auxiliary coordinate variables.
Given a (lat, lon) point, I would like to identify the closest ground pixel on my set of data.
Let's build a 10x10 toy data set:
import numpy as np
import xarray as xr
import cartopy.crs as ccrs
import matplotlib.pyplot as plt
%matplotlib inline
lon, lat = np.meshgrid(np.linspace(-20, 20, 10),
np.linspace(30, 60, 10))
lon += lat/10
lat += lon/10
da = xr.DataArray(data = np.random.normal(0,1,100).reshape(10,10),
dims=['scanline', 'ground_pixel'],
coords = {'lat': (('scanline', 'ground_pixel'), lat),
'lon': (('scanline', 'ground_pixel'), lon)})
ax = plt.subplot(projection=ccrs.PlateCarree())
da.plot.pcolormesh('lon', 'lat', ax=ax, cmap=plt.cm.get_cmap('Blues'),
infer_intervals=True);
ax.scatter(lon, lat, transform=ccrs.PlateCarree())
ax.coastlines()
ax.gridlines(draw_labels=True)
plt.tight_layout()
Note that the lat/lon coordinates identify the centre pixel and the pixel boundaries are automatically inferred by xarray.
Now, say I want to identify the closest ground pixel to Rome.
The best way I came up with so far is to use a scipy's kdtree on a stacked flattened lat/lon array:
from scipy import spatial
pixel_center_points = np.stack((da.lat.values.flatten(), da.lon.values.flatten()), axis=-1)
tree = spatial.KDTree(pixel_center_points)
rome = (41.9028, 12.4964)
distance, index = tree.query(rome)
print(index)
# 36
I have then to apply unravel_index to get my scanline/ground_pixel indexes:
pixel_coords = np.unravel_index(index, da.shape)
print(pixel_coords)
# (3, 6)
Which gives me the scanline/ground_pixel coordinates of the (supposedly) closest ground pixel to Rome:
ax = plt.subplot(projection=ccrs.PlateCarree())
da.plot.pcolormesh('lon', 'lat', ax=ax, cmap=plt.cm.get_cmap('Blues'),
infer_intervals=True);
ax.scatter(da.lon[pixel_coords], da.lat[pixel_coords],
marker='x', color='r', transform=ccrs.PlateCarree())
ax.coastlines()
ax.gridlines(draw_labels=True)
plt.tight_layout()
I'm convinced there must me a much more elegant way to approach this problem. In particular, I would like to get rid of the flattening/unraveling steps (all my attempts to build a kdtree on a two-dimensional array failed miserably), and make use of my xarray dataset's variables as much as possible (adding a new centre_pixel dimension for example, and using it as input to KDTree).

I am going to answer my own question as I believe I came up with a decent solution, which is discussed at much greater length on my blog post on this subject.
Geographical distance
First of all, defining the distance between two points on the earth's surface as simply the euclidean distance between the two lat/lon pairs could lead to inaccurate results depending on the distance between two points. It is thus better to transform the coordinates to ECEF coordinates first and built a KD-Tree on the transformed coordinates. Assuming points on the surface of the planet (h=0) the coordinate transformation is done as such:
def transform_coordinates(coords):
""" Transform coordinates from geodetic to cartesian
Keyword arguments:
coords - a set of lan/lon coordinates (e.g. a tuple or
an array of tuples)
"""
# WGS 84 reference coordinate system parameters
A = 6378.137 # major axis [km]
E2 = 6.69437999014e-3 # eccentricity squared
coords = np.asarray(coords).astype(np.float)
# is coords a tuple? Convert it to an one-element array of tuples
if coords.ndim == 1:
coords = np.array([coords])
# convert to radiants
lat_rad = np.radians(coords[:,0])
lon_rad = np.radians(coords[:,1])
# convert to cartesian coordinates
r_n = A / (np.sqrt(1 - E2 * (np.sin(lat_rad) ** 2)))
x = r_n * np.cos(lat_rad) * np.cos(lon_rad)
y = r_n * np.cos(lat_rad) * np.sin(lon_rad)
z = r_n * (1 - E2) * np.sin(lat_rad)
return np.column_stack((x, y, z))
Building the KD-Tree
We could then build the KD-Tree by transforming our dataset coordinates, taking care of flattening the 2D grid to a one-dimensional sequence of lat/lon tuples. This is because the KD-Tree input data needs to be (N,K), where N is the number of point and K is the dimensionality (K=2 in our case, as we assume no heigth component).
# reshape and stack coordinates
coords = np.column_stack((da.lat.values.ravel(),
da.lon.values.ravel()))
# construct KD-tree
ground_pixel_tree = spatial.cKDTree(transform_coordinates(coords))
Querying the tree and indexing the xarray dataset
Querying the tree is now as simple as transforming our point's lat/lon coordinates to ECEF and passing those to the tree's query method:
rome = (41.9028, 12.4964)
index = ground_pixel_tree.query(transform_coordinates(rome))
In doing so though, we need to unravel our index on the original dataset's shape, to get the scanline/ground_pixel indexes:
index = np.unravel_index(index, self.shape)
We could now use the two components to index our original xarray dataset, but we could also build two indexers to use with xarray pointwise indexing feature:
index = xr.DataArray(index[0], dims='pixel'), \
xr.DataArray(index[1], dims='pixel')
Getting the closest pixel is now easy and elegant at the same time:
da[index]
Note that we could also query more than one point at once, and by building the indexers as above, we could still index the dataset with a single call:
da[index]
Which would then return a subset of the dataset containing the closest ground pixels to our query points.
Further readings
Using the euclidean norm on the lat/lon tuples could be accurate enough for smaller distance (thing of it as approximating the earth as flat, it works on a small scale). More details on geographic distances here.
Using a KD-Tree to find the nearest neighbour is not the only way to address this problem, see this very comprehensive article.
An implementation of KD-Tree directly into xarray is in the pipeline.
My blog post on the subject.

Scale square matrix in geometrical sense using python

I have an matrix (ndarray) with real values that I want to scale in a geometrical sense - that is expand the matrix's size while keeping the values as similar as possible. It can be viewed as scaling an image.
But my matrix is NOT an image. I have real values ranging from 8,000 to 50,000. As far as I know these values cannot represent anything from an usual image point of view.
I have searched the web for answers but every answer suggested using PIL or similar image processing libraries, that use standard pixel values that wouldn't accept my matrix.
So is there a way to scale a matrix containing any real numbers in the geometrical (or image) sense?
Is there a python library for that or list comprehension of some kind or someting similar?
Thank you.

What you're describing is 2D interpolation. Scipy provides an implementation in scipy.interpolate.RectBivariateSpline
from scipy.interpolate import RectBivariateSpline
# sample data
data = np.random.rand(8, 4)
width, height = data.shape
xs = np.arange(width)
ys = np.arange(height)
# target size and interpolation locations
new_width, new_height = width*2, height*2
new_xs = np.linspace(0, width-1, new_width)
new_ys = np.linspace(0, height-1, new_height)
# create the spline object, and use it to interpolate
spline = RectBivariateSpline(xs, ys, data) #, kx=1, ky=1) for linear interpolation
spline(new_xs, new_ys)

trouble with performing coordinate map/interpolation with interp2d

I have what is essentially a 4 column lookup table: cols 1, 2 are the respective xi,yj coordinates which map to x'i, y'j coordinates in the respective 3rd and 4th cols.
My goal is to provide a method to enter some (xnew,ynew) position within the range of my look-up values in the 1st and 2nd columns(xi,yj) then map that position to an interpolated (x'i,y'j) from the range of positions in the 3rd and 4th cols of the lut.
I have tried using interp2d, but have not been able to figure out how to enter the arrays into the proper format. For example: I don't understand why scipy.interpolate.interp2d(x'i, y'j, [xi,yj] kind='linear') gives me the following error:
ValueError: Invalid length for input z for non rectangular grid'.
This seems so simple, but I have not been able to figure it out. I will gladly provide more information if required.

interp2d requires that the interpolated function be 1D, see the docs:
z : 1-D ndarray The values of the function to interpolate at the data
points. If z is a multi-dimensional array, it is flattened before use.
So when you enter [xi,yj], it gets converted from its (2, n) shape to (2*n,), hence the error.
You can get around this setting up two different interpolating functions, one for each coordinate. If your lut is a single array of shape (n, 4), you would do something like:
x_interp = scipy.interpolate.interp2d(lut[0], lut[1], lut[2], kind = 'linear')
y_interp = scipy.interpolate.interp2d(lut[0], lut[1], lut[3], kind = 'linear')
And you can now do things like:
new_x, new_y = x_interp(x, y), y_interp(x, y)