I have measurements (PPI arc scans) taken with a doppler wind lidar. The data is stored in a pandas dataframe where rows represent azimuth angle and columns represent radial distance (input shape = 30x197). Link to example scan, (csv). I want to transform this to a cartesian coordinate system, and output a 2d array which is re-gridded into x,y coordinates instead of polar with the values stored in the appropriate grid cell. Interpolation (nearest neighbor) is ok and so is zero or NaN padding of areas where no data exists.
Ideally the X and Y grid should correspond to the actual distances between points, but right now I'm just trying to get this working. This shouldn’t be terribly difficult, but I’m having trouble obtaining the result I want.
So far, I have working code which plots on a polar axis beautifully (example image) but this won't work for the next steps of my analysis.
I have tried many different approaches with scipy.interpolate.griddata, scipy.ndimage.geometric_transform, and scipy.ndimage.map_coordinates but haven't gotten the correct output. Here is an example of my recent attempt (df_polar is the csv file linked):
# Generate polar and cartesian meshgrids
r = df_polar.columns
theta = df_polar.index
theta = np.deg2rad(theta)
# Polar meshgrid
rad_c, theta_c = np.meshgrid(r,theta)
# Cartesian meshgrid
X = rad_c * np.cos(theta_c)
Y = rad_c * np.sin(theta_c)
x,y = np.meshgrid(X,Y)
# Interpolate from polar to cartesian grid
new_grid = scipy.interpolate.griddata(
(rad_c.flatten(), theta_c.flatten()),
np.array(df_polar).flatten(), (x,y), method='nearest')
The result is not correct at all, and from reading the documentation and examples I don't understand why. I would greatly appreciate any tips on where I have gone wrong. Thanks a lot!!
I think you might be feeding griddata the wrong points. It wants cartesian points and if you want the values interpolated over a regular x/y grid you need to create one and provide that too.
Try this and let me know if it produces the expected result. It's hard for me to tell if this is what it should produce:
from scipy.interpolate import griddata
import pandas as pd
import numpy as np
df_polar = pd.read_csv('onescan.txt', index_col=0)
# Generate polar and cartesian meshgrids
r = pd.to_numeric(df_polar.columns)
theta = np.deg2rad(df_polar.index)
# Polar meshgrid
rad_c, theta_c = np.meshgrid(r, theta)
# Cartesian equivalents of polar co-ordinates
X = rad_c*np.cos(theta_c)
Y = rad_c*np.sin(theta_c)
# Cartesian (x/y) meshgrid
grid_spacing = 100.0 # You can change this
nx = (X.max() - X.min())/grid_spacing
ny = (Y.max() - Y.min())/grid_spacing
x = np.arange(X.min(), X.max() + grid_spacing, grid_spacing)
y = np.arange(Y.min(), Y.max() + grid_spacing, grid_spacing)
grid_x, grid_y = np.meshgrid(x, y)
# Interpolate from polar to cartesian grid
new_grid = griddata(
(X.flatten(), Y.flatten()),
df_polar.values.flatten(),
(grid_x, grid_y),
method='nearest'
)
The resulting values look something like this (with grid_spacing = 10 and flipping x and y):
import matplotlib.pyplot as plt
plt.imshow(new_grid.T, cmap='hot')
Clearly interpolate "nearest" needs taming...
Related
i have a function that calculates a z values based on given x y values in space. I am trying to combine all the data together into a 3D grid however i noticed that the z values are not mapping correctly. In other words when print the xyz and perform the calculation as a check in excel I do not get the right z values, but i'm confident my function is calculating correctly. If i check it on an individual basis it gives the result i'm looking for. So i'm pretty sure the z values are getting mapped to the incorrect x,y.
FYI the reason i need the grid together as XYZ is: once i get the function running i need to perform grid math on the resulting grid. For example i need to find certain locations based on a given X and Y and then find nodes that correspond to a certain z value and sum the area of the nodes...etc. I haven't gotten there yet obviously. I'm new to python and working my way there.
What am I doing wrong here? Note I don't get any errors.
Any help is greatly appreciated.
What am i doing wrong?
import pandas as pd
import math
import numpy as np
import matplotlib.mlab as mlab
import matplotlib.tri as tri
import matplotlib.pyplot as plt
from matplotlib import rcParams
#define the flow potential equation where X,Y is the injection well locations and x,y is the point of interest, Q is flow rate
def func(X, Y, x, y, Q):
return (Q / (2 * np.pi)) * np.arctan((y-Y)/(x-X))
# necessary data
X1=2318743.658
Y1=797346.704
Q1=5
X2=2318690.718
Y2=797343.693
Q2=5
X3=2318715.221
Y3=797309.685
Q3=5
#initiate the XY grid - this will be a standard that will encompass all IW and MW
xi = np.linspace(2318675,2318800,625)
yi = np.linspace(797300,797375,375)
#mesh the grid in to x,y space
x,y = np.meshgrid(xi,yi)
#calculate the valus over the grid at every x,y using the defined function above
zi = (func(X1,Y1,x,y,Q1)+func(X2,Y2,x,y,Q2)+func(X3,Y3,x,y,Q3))
#reshape the xy space into 3d space
xy = np.array([[(x, y) for y in yi] for x in xi])
#reshape z into 3d space
z = np.array(zi).reshape(xy.shape[0],xy.shape[1], -1)
#combined xyz into a single grid
xyz = np.concatenate((xy, z), axis = -1)
I believe I was able to resolve the issue by changing this line of code:
xy = np.array([[(x, y) for y in yi] for x in xi])
to this:
xy = np.array([[(x, y) for x in xi] for y in yi])
I have a two-dimensional array that represents function values at positions in a polar coordinate system. For example:
import numpy as np
radius = np.linspace(0, 1, 50)
angle = np.linspace(0, 2*np.pi, radius.size)
r_grid, a_grid = np.meshgrid(radius, angle)
data = np.sqrt((r_grid/radius.max())**2
+ (a_grid/angle.max())**2)
Here the data is arranged in a rectangular grid corresponding to the polar coordinates. I want to rearrange the data in the array such that the axes represent the corresponding Cartesian coordinate system. The old versus new layout can be visualized as follows:
import matplotlib.pyplot as plt
fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=plt.figaspect(0.5))
ax1.set(title='Polar coordinates', xlabel='Radius', ylabel='Angle')
ax1.pcolormesh(r_grid, a_grid, data)
ax2.set(title='Cartesian coordinates', xlabel='X', ylabel='Y')
x_grid = r_grid * np.cos(a_grid)
y_grid = r_grid * np.sin(a_grid)
ax2.pcolormesh(x_grid, y_grid, data)
Here the coordinates are explicitly given and the plot is adjusted accordingly. I want the data to be rearranged in the data array itself instead. It should contain all values, optionally filling with zeros to fit the shape (similar to scipy.ndimage.rotate(..., reshape=True)).
If I manually loop over the polar arrays to compute the Cartesian coordinates, the result contains empty regions which ideally should be filled as well:
new = np.zeros_like(data)
visits = np.zeros_like(new)
for r, a, d in np.nditer((r_grid, a_grid, data)):
i = 0.5 * (1 + r * np.sin(a)) * new.shape[0]
j = 0.5 * (1 + r * np.cos(a)) * new.shape[1]
i = min(int(i), new.shape[0] - 1)
j = min(int(j), new.shape[1] - 1)
new[i, j] += d
visits[i, j] += 1
new /= np.maximum(visits, 1)
ax2.imshow(new, origin='lower')
Is there a way to achieve the transformation while avoiding empty regions in the resulting data array?
tl;dr: No, not without changing some conditions of your problem.
The artefact you are seeing is a property of the transformation.
It is not due to the fixed resolution in angle for all radii.
Hence, it is not due to a wrong or bad implementation of the transformation.
The Cartesian grid simply implies a higher special resolution at these areas as there are resolved points from the polar map.
The only "clean" way (that I can think of right now) to handle this is to have an adjustable resolution in the polar coordinates to account for the 1/r scaling. (If you input data allows it)
A somewhat cheating way of visualizing this without the gaps would to randomly distribute them over the gaps. The argument here is, that you do not have the resolution to decide in which bin they were to begin with. So you could just randomly throw them in one which might have been a possible origin and not throw them all in same one (as you are doing right now).
However, I would like discourage this stronlgy. It just gives you a prettier plot.
Note, that this is somewhat equivalent to the behaviour of the upper right plot in your question.
This doesn't really give the expected result, but maybe will help you in achieving a solution after some needed correction...
import numpy as np
radius = np.linspace(0, 1, 50)
angle = np.linspace(0, 2*np.pi, radius.size)
r_grid, a_grid = np.meshgrid(radius, angle)
data = np.sqrt((r_grid/radius.max())**2
+ (a_grid/angle.max())**2)
def polar_to_cartesian(data):
new = np.zeros_like(data) * np.nan
x = np.linspace(-1, 1, new.shape[1])
y = np.linspace(-1, 1, new.shape[0])
for i in range(new.shape[0]):
for j in range(new.shape[1]):
x0, y0 = x[j], y[i]
r, a = np.sqrt(x0**2 + y0**2), np.arctan2(y0, x0)
data_i = np.argmin(np.abs(a_grid[:, 0] - a))
data_j = np.argmin(np.abs(r_grid[0, :] - r))
val = data[data_i, data_j]
if r <= 1:
new[i, j] = val
return new
new = polar_to_cartesian(data)
fig, ax = plt.subplots()
ax.imshow(new, origin='lower')
EDIT:
Modified using np.arctan2 according to the suggestions of OP.
You could loop over the Cartesian array, transforming each grid point to polar coordinates and approximating the function value by interpolation from your polar grid data. You may still want to leave the corner regions blank though, for lack of close enough data.
I don't think there is a better way, unless of course you have access to the original function.
I'm trying to find a method of linear interpolation in 2D over a regular grid using python, but each proposed type in scipy seems to have it's disadvantages.
My aim is basically:
Have smooth linearly interpolated data over a regular grid, or as close as possible
The original data can be at arbitrary locations
(optional) Linearly extrapolate to the edges
But all the functions seem to have problems with this:
Functions like griddata, interp2d, LinearNDInterpolator appear to create triangles and interpolate within them, creating a bunch of hard lines/creases that I can't have.
Rbf seems at first to do exactly what I want, but when presented with planes that are flat, it generates an interpolation based on some kind of sphere, creating a curved surface.
If Rbf would simply interpolate a flat plane as a flat plane using the linear setting, as would be expected, it'd be perfect.
Are there any ideas on how to achieve this, or if there's another function that does what I'm after? I've attached a basic example below.
import numpy as np
from scipy import interpolate
import matplotlib.pyplot as plt
#create some objects to store data
x=np.empty((2,2))
y=np.empty((2,2))
f_shape=(100,100,100)
#generate coordinates
x[0, :] = 0
x[-1, :] = f_shape[0]
y[:, 0] = 0
y[:, -1] = f_shape[1]
#--------constant height----------
z=np.full((2,2),50)
#create interpolation function and interpolate across grid
interp=interpolate.Rbf(x,y,z,function='linear')
grid=np.mgrid[0:f_shape[0],0:f_shape[1]]
result=interp(grid[0,:,:],grid[1,:,:])
plt.imshow(result) #incorrect curved surface from constant height!!!
#--------random heights-----------
z=np.random.uniform(25,75,(2,2))
#create interpolation function and interpolate across grid
interp=interpolate.Rbf(x,y,z,function='linear')
grid=np.mgrid[0:f_shape[0],0:f_shape[1]]
result=interp(grid[0,:,:],grid[1,:,:])
plt.imshow(result) #Apparently nice smooth linear-ish interpolation
Incorrect curved surface from constant height:
Apparently nice smooth linear-ish interpolation:
I've managed to write a function that suits my purpose. It interpolates (fills in) a plane from a grid of coordinates by interpolating along the grid lines, then interpolating the plane in the x and y directions, and taking the average of the two.
It should be possible to speed this up a bit by reshaping the coordinates into a 1D vector, interpolating the plane in one go, then reshaping back into 2D. However this code is certainly fast enough already for reasonable plane sizes.
Seems to work ok if the coordinates are outside of the plane too.
Extrapolation also works if the grid is approximately regular. It'll extrapolate regardless but you'll start to see some sharp creases away from the edge as the grid irregularity increases.
Here's the code. An example is provided in the docstring.
def interlin2d(x,y,z,fsize):
"""
Linear 2D interpolation of a plane from arbitrary gridded points.
:param x: 2D array of x coordinates
:param y: 2D array of y coordinates
:param z: 2D array of z coordinates
:param fsize: Tuple of x and y dimensions of plane to be interpolated.
:return: 2D array with interpolated plane.
This function works by interpolating lines along the grid point in both dimensions,
then interpolating the plane area in both the x and y directions, and taking the
average of the two. Result looks like a series of approximately curvilinear quadrilaterals.
Note, the structure of the x,y,z coordinate arrays are such that the index of the coordinates
indicates the relative physical position of the point with respect to the plane to be interpoalted.
Plane is allowed to be a subset of the range of grid coordinates provided.
Extrapolation is accounted for, however sharp creases will start to appear
in the extrapolated region as the grid of coordinates becomes increasingly irregular.
Scipy's interpolation function is used for the grid lines as it allows for proper linear extrapolation.
However Numpy's interpolation function is used for the plane itself as it is robust against gridlines
that overlap (divide by zero distance).
Example:
#set up number of grid lines and size of field to interpolate
nlines=[3,3]
fsize=(100,100,100)
#initialize the coordinate arrays
x=np.empty((nlines[0],nlines[1]))
y=np.empty((nlines[0],nlines[1]))
z=np.random.uniform(0.25*fsize[2],0.75*fsize[2],(nlines[0],nlines[1]))
#set random ordered locations for the interior points
spacings=(fsize[0]/(nlines[0]-2),fsize[1]/(nlines[1]-2))
for k in range(0, nlines[0]):
for l in range(0, nlines[1]):
x[k, l] = round(random.uniform(0, 1) * (spacings[0] - 1) + spacings[0] * (k - 1) + 1)
y[k, l] = round(random.uniform(0, 1) * (spacings[1] - 1) + spacings[1] * (l - 1) + 1)
#fix the edge points to the edge
x[0, :] = 0
x[-1, :] = fsize[1]-1
y[:, 0] = 0
y[:, -1] = fsize[0]-1
field = interlin2d(x,y,z,fsize)
"""
from scipy.interpolate import interp1d
import numpy as np
#number of lines in grid in x and y directions
nsegx=x.shape[0]
nsegy=x.shape[1]
#lines along the grid points to be interpolated, x and y directions
#0 indicates own axis, 1 is height (z axis)
intlinesx=np.empty((2,nsegy,fsize[0]))
intlinesy=np.empty((2,nsegx,fsize[1]))
#account for the first and last points being fixed to the edges
intlinesx[0,0,:]=0
intlinesx[0,-1,:]=fsize[1]-1
intlinesy[0,0,:]=0
intlinesy[0,-1,:]=fsize[0]-1
#temp fields for interpolation in x and y directions
tempx=np.empty((fsize[0],fsize[1]))
tempy=np.empty((fsize[0],fsize[1]))
#interpolate grid lines in the x direction
for k in range(nsegy):
interp = interp1d(x[:,k], y[:,k], kind='linear', copy=False, fill_value='extrapolate')
intlinesx[0,k,:] = np.round(interp(range(fsize[0])))
interp = interp1d(x[:, k], z[:, k], kind='linear', copy=False, fill_value='extrapolate')
intlinesx[1, k, :] = interp(range(fsize[0]))
intlinesx[0,:,:].sort(0)
# interpolate grid lines in the y direction
for k in range(nsegx):
interp = interp1d(y[k, :], x[k, :], kind='linear', copy=False, fill_value='extrapolate')
intlinesy[0, k, :] = np.round(interp(range(fsize[1])))
interp = interp1d(y[k, :], z[k, :], kind='linear', copy=False, fill_value='extrapolate')
intlinesy[1, k, :] = interp(range(fsize[1]))
intlinesy[0,:,:].sort(0)
#interpolate plane in x direction
for k in range(fsize[1]):
tempx[k, :] = np.interp(range(fsize[1]),intlinesx[0,:,k], intlinesx[1,:,k])
#interpolate plane in y direction
for k in range(fsize[1]):
tempy[:, k] = np.interp(range(fsize[0]), intlinesy[0, :, k], intlinesy[1, :, k])
return (tempx+tempy)/2
Example of interpolation based on 9 points (shown as red dots)
Scipy's griddata works just fine:
import numpy as np
from scipy import interpolate as intp
import matplotlib.pyplot as plt
%matplotlib inline
grid_size = G = 100
height = H = 50
points = np.array([
(0, 0),
(G-1, 0),
(0, G-1),
(G-1, G-1)
], dtype=np.float32)
gy, gx = np.mgrid[:G, :G]
result = intp.griddata(points, np.full(points.shape[0], H), (gy, gx))
And the plot:
plt.imshow(result, interpolation='none')
plt.colorbar()
And just to be sure:
>>> np.allclose(result, 50)
True
I want to extract given point probes from a surface contour plot. Therefore I create the surface with following code snippet:
def createInterpolatedSurface():
npts=1000
xi = np.linspace(min(x), max(x),npts)
yi = np.linspace(min(y), max(y),npts)
xi,yi=np.meshgrid(xi,yi,indexing='xy')
ui=scipy.interpolate.griddata((x,y),u,(xi,yi),method='linear')
return xi,yi,ui
My next step is to set up an array of points, with initSensorArray as nested Loop function, I am interested in:
So there is my main problem. I want to use the realistic physical coordinates of my points, not the ij indexing coordinates of the griddata interpolation function.
For example: Point(0.5,0.1) in physical space equals Pointg(100,125) in griddata ij indexing,.
How can I map the physical point coordinates to the griddata, extrapolate the points and map them back?
Thanks for help
You can use scipy'` interpolate functions in 2D,
from scipy import interpolate
x,y,u = createInterpolatedSurface()
#Create an interpolation function
f = interpolate.interp2d(x, y, u, kind='cubic')
#Use interpolation function to get results
xpoint = 0.5
ypoint = 0.1
upoint = f(xpoint, ypoint)
I have a set of data given here where in the first and second columns there are the sky coordinates (ra,dec), respectively and in the third and forth, the coordinates in a Cartesian system (x,y).
I need to make a two-dimensional interpolation surface using coordinates x and y and another using Ra and Dec. The problem is the existence of masked regions, as shown in the figure above. I can illustrate the missing data just by plotting them (There is non NaN value in the catalogue). That is what I so far tried and didn't give the right answer:
from scipy.interpolate import griddata
import numpy as np
import matplotlib.pyplot as plt
data = np.loadtxt('test.asc')
ra = data[:,0]
dec = data[:,1]
Xpos = data[:,2]
Ypos = data[:,3]
xi = np.linspace(Xpos.min(), Xpos.max(), 1000)
yi = np.linspace(Ypos.min(), Ypos.max(), 1000)
xi, yi = np.meshgrid(xi, yi, copy=False)
ra_int = griddata(data[:,2:4], ra, (xi.flatten(), yi.flatten()),
method='cubic')
dec_int = griddata(data[:,2:4], dec, (xi.flatten(), yi.flatten()),
method='cubic')
Using griddata fails and return just NaN values. Is there any way to do this interpolation in order to estimate the values of Ra and Dec from a given x and y coordinates even in the masked regions (map from x and y to ra and dec)?
If I get it right then it is like this:
just shift the Cartesian coordinate system to middle of the CCD and also the Equatoreal coordinates to middle of CCD. Then compute x,y separately. The only thing you need is to compute focus length f separately for x and y !!!
pos is the cartesian coordinate (x or y)
ang is the equatoreal coordinate (RA or Dec)
get edge point from the database
shift the angles to middle of CCD
compute focus (fx,fy) from it
f = pos/tan(ang)
now you can compute the projection for any entry in dataset
shift the angles to middle of CCD then compute x,y by
pos=f*tan(ang)
shift back from CCD middle to original Cartesian coordinates. You should check few points if is this approach correct
[notes]
x axis is mirrored in your output so just use x=-x at the end before shifting back to original Cartesian coordinates or leave focus f negative.
if your CCD is not axis aligned to equator then you need to compute the rotation (angle between X axis and equator) and apply rotation around Z axis after conversion before shifting back...