Develop Reference

Regrid 2D data onto larger 2D grid at given coordinates in Python

I have a square 2D array data that I would like to add to a larger 2D array frame at some given set of non-integer coordinates coords. The idea is that data will be interpolated onto frame with it's center at the new coordinates.
Some toy data:
# A gaussian to add to the frame
x, y = np.meshgrid(np.linspace(-1,1,10), np.linspace(-1,1,10))
data = 50*np.exp(-np.sqrt(x**2+y**2)**2)
# The frame to add the gaussian to
frame = np.random.normal(size=(100,50))
# The desired (x,y) location of the gaussian center on the new frame
coords = 23.4, 22.6
Here's the idea. I want to add this:
to this:
to get this:
If the coordinates were integers (indexes), of course I could simply add them like this:
frame[23:33,22:32] += data
But I want to be able to specify non-integer coordinates so that data is regridded and added to frame.
I've looked into PIL.Image methods but my use case is just for 2D data, not images. Is there a way to do this with just scipy? Can this be done with interp2d or a similar function? Any guidance would be greatly appreciated!

Scipy's shift function from scipy.ndimage.interpolation is what you are looking for, as long as the grid spacings between data and frame overlap. If not, look to the other answer. The shift function can take floating point numbers as input and will do a spline interpolation. First, I put the data into an array as large as frame, then shift it, and then add it. Make sure to reverse the coordinate list, as x is the rightmost dimension in numpy arrays. One of the nice features of shift is that it sets to zero those values that go out of bounds.
import numpy as np
import matplotlib.pyplot as plt
from scipy.ndimage.interpolation import shift
# A gaussian to add to the frame.
x, y = np.meshgrid(np.linspace(-1,1,10), np.linspace(-1,1,10))
data = 50*np.exp(-np.sqrt(x**2+y**2)**2)
# The frame to add the gaussian to
frame = np.random.normal(size=(100,50))
x_frame = np.arange(50)
y_frame = np.arange(100)
# The desired (x,y) location of the gaussian center on the new frame.
coords = np.array([23.4, 22.6])
# First, create a frame as large as the frame.
data_large = np.zeros(frame.shape)
data_large[:data.shape[0], :data.shape[1]] = data[:,:]
# Subtract half the distance as the bottom left is at 0,0 instead of the center.
# The shift of 4.5 is because data is 10 points wide.
# Reverse the coords array as x is the last coordinate.
coords_shift = -4.5
data_large = shift(data_large, coords[::-1] + coords_shift)
frame += data_large
# Plot the result and add lines to indicate to coordinates
plt.figure()
plt.pcolormesh(x_frame, y_frame, frame, cmap=plt.cm.jet)
plt.axhline(coords[1], color='w')
plt.axvline(coords[0], color='w')
plt.colorbar()
plt.gca().invert_yaxis()
plt.show()
The script gives you the following figure, which has the desired coordinates indicated with white dotted lines.

One possible solution is to use scipy.interpolate.RectBivariateSpline. In the code below, x_0 and y_0 are the coordinates of a feature from data (i.e., the position of the center of the Gaussian in your example) that need to be mapped to the coordinates given by coords. There are a couple of advantages to this approach:
If you need to "place" the same object into multiple locations in the output frame, the spline needs to be computed only once (but evaluated multiple times).
In case you actually need to compute integrated flux of the model over a pixel, you can use the integral method of scipy.interpolate.RectBivariateSpline.
Resample using spline interpolation:
from scipy.interpolate import RectBivariateSpline
x = np.arange(data.shape[1], dtype=np.float)
y = np.arange(data.shape[0], dtype=np.float)
kx = 3; ky = 3; # spline degree
spline = RectBivariateSpline(
x, y, data.T, kx=kx, ky=ky, s=0
)
# Define coordinates of a feature in the data array.
# This can be the center of the Gaussian:
x_0 = (data.shape[1] - 1.0) / 2.0
y_0 = (data.shape[0] - 1.0) / 2.0
# create output grid, shifted as necessary:
yg, xg = np.indices(frame.shape, dtype=np.float64)
xg += x_0 - coords[0] # see below how to account for pixel scale change
yg += y_0 - coords[1] # see below how to account for pixel scale change
# resample and fill extrapolated points with 0:
resampled_data = spline.ev(xg, yg)
extrapol = (((xg < -0.5) | (xg >= data.shape[1] - 0.5)) |
((yg < -0.5) | (yg >= data.shape[0] - 0.5)))
resampled_data[extrapol] = 0
Now plot the frame and resampled data:
plt.figure(figsize=(14, 14));
plt.imshow(frame+resampled_data, cmap=plt.cm.jet,
origin='upper', interpolation='none', aspect='equal')
plt.show()
If you also want to allow for scale changes, then replace code for computing xg and yg above with:
coords = 20, 80 # change coords to easily identifiable (in plot) values
zoom_x = 2 # example scale change along X axis
zoom_y = 3 # example scale change along Y axis
yg, xg = np.indices(frame.shape, dtype=np.float64)
xg = (xg - coords[0]) / zoom_x + x_0
yg = (yg - coords[1]) / zoom_y + y_0
Most likely this is what you actually want based on your example. Specifically, the coordinates of pixels in data are "spaced" by 0.222(2) distance units. Therefore it actually seems that for your particular example (whether accidental or intentional), you have a zoom factor of 0.222(2). In that case your data image would shrink to almost 2 pixels in the output frame.
Comparison to #Chiel answer
In the image below, I compare the results from my method (left), #Chiel's method (center) and difference (right panel):
Fundamentally, the two methods are quite similar and possibly even use the same algorithm (I did not look at the code for shift but based on the description - it also uses splines). From comparison image it is visible that the biggest differences are at the edges and, for unknown to me reasons, shift seems to truncate the shifted image slightly too soon.
I think the biggest difference is that my method allows for pixel scale changes and it also allows re-use of the same interpolator to place the original image at different locations in the output frame. #Chiel's method is somewhat simpler but (what I did not like about it is that) it requires creation of a larger array (data_large) into which the original image is placed in the corner.

While the other answers have gone into detail, but here's my lazy solution:
xc,yc = 23.4, 22.6
x, y = np.meshgrid(np.linspace(-1,1,10)-xc%1, np.linspace(-1,1,10)-yc%1)
data = 50*np.exp(-np.sqrt(x**2+y**2)**2)
frame = np.random.normal(size=(100,50))
frame[23:33,22:32] += data
And it's the way you liked it. As you mentioned, the coordinates of both are the same, so the origin of data is somewhere between the indices. Now just simply shift it by the amount you want it to be off a grid point (remainder to one) in the second line and you're good to go (you might need to flip the sign, but I think this is correct).

Why does gdal_grid turn image upside-down?

I'm trying to use gdal_grid to make an elevation grid from a surface in a geojson. I use this command:
gdal_grid -a linear:radius=0 inputSurface.geojson outputFile.tif
It seems to give the correct pixel values, but if I open the result in Global Mapper or QGIS, the image is flipped/mirrored in a horizontal axis, such that the tif is directly below the surface and upside-down.
What is the reason for this and how do I fix it??
Update
I already tried changing the geotransform, but it hasn't totally fixed my problem.
I looked at the resulting image in gdalinfo and found out that the upper left corner is actually the lower left corner, so I set it using the SetGeoTransform. This moved it to the correct location, but it is still upside-down. (This may by dependent on the projection, which might cause problems later)
I also tried looking at the pixel width in the geotransform as mentioned below:
Xgeo = GT[0] + Xpixel*GT[1] + Yline*GT[2]
Ygeo = GT[3] + Xpixel*GT[4] + Yline*GT[5]
The image returned by gdal_grid has a positive GT[5], but unfortunately changing it to -GT[5] doesn't change anything.
The code I used to change the geotransform:
transform = list(ds.GetGeoTransform())
transform = [upperLeftX, transform[1], 0, upperLeftY, 0, -transform[5]]
ds.SetGeoTransform(transform)

GDAL's georeferencing is commonly specified by two sets of parameters. The first is the spatial reference, which defines the coordinate system (UTM, WGS, something more localized). The spatial reference for a raster is set using gdal.Dataset.setProjection(). The second piece of georeferencing is the GeoTransform, which translates (row, column) pixel indices into coordinates in the coordinate system. It is likely the geotransform that you need to update to make your image "unflipped".
The GeoTransform is a tuple of 6 values, which relate raster indices into coordinates.
Xgeo = GT[0] + Xpixel*GT[1] + Yline*GT[2]
Ygeo = GT[3] + Xpixel*GT[4] + Yline*GT[5]
Because these are raster images, the (line, pixel) or (row, col) coordinates start from the top left of the image.
[ ]----> column
|
|
v row
This means that GT[1] will be positive when the image is positioned "upright" in the coordinate system. Similarly, and sometimes counter-intuitively, GT[5] will be negative because the y value should decrease for every increasing row in the image. This isn't a requirement, but it is very common.
Modifying the GeoTransform
You state that the image is upside down and below where is should be. This isn't guaranteed to be a fix, but it will get you started. It's easier if you have the image in front of you and can experiment or compare coordinates...
import gdal
# open dataset as readable/writable
ds = gdal.Open('input.tif', gdal.GA_Update)
# get the GeoTransform as a tuple
gt = gdal.GetGeoTransform()
# change gt[5] to be it's negative, flipping the image
gt_new = (gt[0], gt[1], gt[2], gt[3], gt[4], -1 * gt[5])
# set the new GeoTransform, effectively flipping the image
ds.SetGeoTransform(gt_new)
# delete the dataset reference, flushing the cache of changes
del ds

I ended up having more problems with gdal_grid, where it just crashes at seemingly random places, so I'm using the scipy.interpolate-function called griddata in stead. This uses a meshgrid to get the coordinates in the grid, and I had to tile it up because of memory restrictions of meshgrid.
import scipy.interpolate as il #for griddata
import numpy as np
# meshgrid of coords in this tile
gridX, gridY = np.meshgrid(xi[c*tcols:(c+1)*tcols], yi[r*trows:(r+1)*trows][::-1])
## Creating the DEM in this tile
zi = il.griddata((coordsT[0], coordsT[1]), coordsT[2], (gridX, gridY),method='linear',fill_value = nodata) # fill_value to prevent NaN at polygon outline
raster.GetRasterBand(1).WriteArray(zi,c*tcols,nrows-r*trows-rtrows)
The linear interpolation seems to do the same as gdal_grid is supposed to. This was actually effected by making the 5'th element in the geotransform negative as described in the question update.
See description at scipy.interpolate.griddata.
A few things to note:
The point used in the geotransform should be upper-left
The resolution in y-direction should be negative
In the projection (at least the ones I use) positive y-direction is up
In numpy arrays positive y-direction is down
When using gdal's WriteArray it uses the upper left corner
Hope this helps other people's confusion.

I've solved a similar issue by simply re-projecting the results of the gdal_grid. Give this a try (replacing the epsg code with your projection and replacing the input/output filepaths):
gdalwarp -s_srs epsg:4326 -t_srs epsg:4326 gdal_grid_result.tif inverted_output.tif

it does not. it is simply the standards of the tool rendering it. try opening it in QGIS and youll notice it is right side up.

Why are Basemap south polar stereographic map projection coordinates not agreeing with those of data sets in the same projection?

Some satellite based earth observation products provide latitude/longitude information while others provide the X/Y coordinates within a given grid projection (and there are also some having both, see example).
My approach in the second case is to set up a Basemap map which has the same parameters (projection, ellipsoid, origin of map) as given by the data provider in a way that the given X/Y values equal the Basemap coordinates. However if I do so the geolocation does not agree with other data sets including the Basemap coastline.
I have experienced this with three different data sets from different trustworthy sources. For the minimal example I use Landsat data provided by the U.S. Geological Survey which includes both, X/Y coordinates of a South Polar Stereographic grid and the corresponding lat/lon coordinates for all four corners of the image.
From a Landsat metafile we get (ID: LC82171052016079LGN00):
CORNER_UL_LAT_PRODUCT = -66.61490 CORNER_UL_LON_PRODUCT = -61.31816
CORNER_UR_LAT_PRODUCT = -68.74325 CORNER_UR_LON_PRODUCT = -58.04533
CORNER_LL_LAT_PRODUCT = -67.68721 CORNER_LL_LON_PRODUCT = -67.01109
CORNER_LR_LAT_PRODUCT = -69.94052 CORNER_LR_LON_PRODUCT = -64.18581
CORNER_UL_PROJECTION_X_PRODUCT = -2259300.000
CORNER_UL_PROJECTION_Y_PRODUCT = 1236000.000
CORNER_UR_PROJECTION_X_PRODUCT = -1981500.000
CORNER_UR_PROJECTION_Y_PRODUCT = 1236000.000
CORNER_LL_PROJECTION_X_PRODUCT = -2259300.000
CORNER_LL_PROJECTION_Y_PRODUCT = 958500.000
CORNER_LR_PROJECTION_X_PRODUCT = -1981500.000
CORNER_LR_PROJECTION_Y_PRODUCT = 958500.000
...
GROUP = PROJECTION_PARAMETERS MAP_PROJECTION = "PS" DATUM = "WGS84"
ELLIPSOID = "WGS84" VERTICAL_LON_FROM_POLE = 0.00000 TRUE_SCALE_LAT =
-71.00000 FALSE_EASTING = 0 FALSE_NORTHING = 0 GRID_CELL_SIZE_PANCHROMATIC = 15.00 GRID_CELL_SIZE_REFLECTIVE = 30.00
GRID_CELL_SIZE_THERMAL = 30.00 ORIENTATION = "NORTH_UP"
RESAMPLING_OPTION = "CUBIC_CONVOLUTION" END_GROUP =
PROJECTION_PARAMETERS
By using Basemap with the right map projection we should be able to derive the corner lat/lon values from the X/Y values:
import numpy as np
from mpl_toolkits.basemap import Basemap
m=Basemap(resolution='h',projection='spstere', ellps='WGS84', boundinglat=-60,lon_0=180, lat_ts=-71)
x_crn=np.array([-2259300,-1981500,-2259300,-1981500])# upper left, upper right, lower left, lower right
y_crn=np.array([1236000, 1236000, 958500, 958500])# upper left, upper right, lower left, lower right
x0, y0= m(0, -90)
#Basemap coordinates at the south pole
#note that (0,0) of the Basemap is in a corner of the map,
#while other data sets use the south pole.
#This is easy to take into account:
lon_crn, lat_crn = m(x0-x_crn, y0-y_crn, inverse=True)
print 'lon_crn: '+str(lon_crn)
print 'lat_crn: '+str(lat_crn)
Which returns:
lon_crn: [-61.31816102 -58.04532791 -67.01108782 -64.1858106 ]
lat_crn: [-67.23548626 -69.3099076 -68.28071626 -70.47651326]
As you can see the longitudes agree to the given precision with those from the metafile, but the latitudes are to low.
I can approximate the latitudes by:
lat_crn=(lat_crn+90.)*1.0275-90.
But this is really not satisfying.
This is how the image is located if using the X/Y corner coordinates from the metafile (in red the Basemap drawcoastlines()):
and this is how it looks like using the corner lat/lon:
In this case I can simply use the lat/lon coordinates, but as mentioned before there are datasets (like this) which is provided by X/Y coordinates only, which makes it very important to rely on the Basemap projection. I know that there are other modules to re-project the data as a potential workaround, but it should work without other modules and a re-projection could introduce errors itself.
As this problem appears with different data sets I like to believe that it is a bug in the Basemap module, but I might also make the same mistake again and again or have wrong expectations.

I did some experimentation and it seems like changing lat_ts has no effect with projection='spstere'. In fact, it seems as if the projection latitude is implicitly assumed to be lat_ts=-90. regardless of what value you assign.
I had more success using projection='stere' instead, so that you would construct the Basemap in your example as follows:
m=Basemap(width=5400000., height=5400000., projection='stere',
ellps='WGS84', lon_0=180., lat_0=-90., lat_ts=-71.)
You may prefer to set the latitude and longitude of the corners instead of the width and height of the plot for your application.

Outline a region in a graph

I have two 2D numpy arrays (of the same dimensions) that I am plotting using matplotlib. The first array I've plotted as a color map in gray-scale. The second one represents an aperture, but it is an irregular shape (some of the pixels get outlined, and it is a set of horizontal and vertical lines that form the outline). I am not sure how to ask it to plot this second array. The array is composed of three numbers (0, 1, and 3), and I only need the pixels of one value (3) to be outlined, but I need the outline to encompass the region of these pixels, not the pixels individually. I need the interior of all the pixels to remain transparent so that I can see the gray-scale color map through it.
Does anyone know how to accomplish this?

That is an interesting question, if I understood it correctly. In order to make sure what you mean, you would like to draw a line with some color around all contiguous areas where the pixel value is 3.
I do not think there is a ready-made function for that, but let's not let that stop us. We will need to create our own function.
We can start by creating a boolean map of the area which needs to be outlined:
import numpy as np
import matplotlib.pyplot as plt
# our image with the numbers 1-3 is in array maskimg
# create a boolean image map which has trues only where maskimg[x,y] == 3
mapimg = (maskimg == 3)
# a vertical line segment is needed, when the pixels next to each other horizontally
# belong to diffferent groups (one is part of the mask, the other isn't)
# after this ver_seg has two arrays, one for row coordinates, the other for column coordinates
ver_seg = np.where(mapimg[:,1:] != mapimg[:,:-1])
# the same is repeated for horizontal segments
hor_seg = np.where(mapimg[1:,:] != mapimg[:-1,:])
# if we have a horizontal segment at 7,2, it means that it must be drawn between pixels
# (2,7) and (2,8), i.e. from (2,8)..(3,8)
# in order to draw a discountinuous line, we add Nones in between segments
l = []
for p in zip(*hor_seg):
l.append((p[1], p[0]+1))
l.append((p[1]+1, p[0]+1))
l.append((np.nan,np.nan))
# and the same for vertical segments
for p in zip(*ver_seg):
l.append((p[1]+1, p[0]))
l.append((p[1]+1, p[0]+1))
l.append((np.nan, np.nan))
# now we transform the list into a numpy array of Nx2 shape
segments = np.array(l)
# now we need to know something about the image which is shown
# at this point let's assume it has extents (x0, y0)..(x1,y1) on the axis
# drawn with origin='lower'
# with this information we can rescale our points
segments[:,0] = x0 + (x1-x0) * segments[:,0] / mapimg.shape[1]
segments[:,1] = y0 + (y1-y0) * segments[:,1] / mapimg.shape[0]
# and now there isn't anything else to do than plot it
plt.plot(segments[:,0], segments[:,1], color=(1,0,0,.5), linewidth=3)
Let us test this by generating some data and showing it:
image = np.cumsum(np.random.random((20,20))-.5, axis=1)
maskimg = np.zeros(image.shape, dtype='int')
maskimg[image > 0] = 3
x0 = -1.5
x1 = 1.5
y0 = 2.3
y1 = 3.8
plt.figure()
plt.imshow(maskimg, origin='lower', extent=[x0,x1,y0,y1], cmap=plt.cm.gray, interpolation='nearest')
plt.axis('tight')
After that we run the procedure on the top, and get:
The code can be made much denser, if needed, but now comments take a lot of space. With large images it might be wise to optimize the image segment creation by finding continuous paths. That will reduce the number of points to plot by a factor of up to three. However, doing that requires a bit different code, which is not as clear as this one. (If there will appear comments asking for that and an appropriate number of upvotes, I'll add it :)

Possible to use a custom arrow or polygon as a marker to plot location and heading in matplotlib?

I have a series of x,y coordinates and associated heading angles for multiple aircraft. I can plot the paths flown, and I would like to use a special marker to mark a particular location along the path that also shows the aircraft's heading when it was at that location.
Using matplotlib.pyplot I've used an arrowhead with no base to do this, but having to define the head and tail locations ended up with inconsistent arrowhead lengths when plotting multiple aircraft. I also used a custom three-sided symbol with the tuple (numsides, style, angle) as well as the wedge and bigvee symbols, but they never look very good.
From Custom arrow style for matplotlib, pyplot.annotate Saullo Castro showed a nice custom arrow (arrow1) that I'm wondering whether it can be used or converted in such a way as to just simply plot it at a given x,y and have its orientation defined by a heading angle.
I can plot the custom arrow with the following. Any ideas on how to rotate it to reflect a heading?
a1 = np.array([[0,0],[0,1],[-1,2],[3,0],[-1,-2],[0,-1],[0,0]], dtype=float)
polB = patches.Polygon(a1, closed=True, facecolor='grey')
ax.add_patch(polB)
Thanks in advance.

So I made the polygon a little simpler and also found that the rotation could be done by using mpl.transforms.Affine2D().rotate_deg_around():
a2 = np.array([[newX,newY+2],[newX+1,newY-1],[newX,newY],[newX-1,newY-1],[newX,newY+2]], dtype=float)
polB = patches.Polygon(a2, closed=True, facecolor='gold')
t2 = mpl.transforms.Affine2D().rotate_deg_around(newX,newY,heading) + newax.transData
polB.set_transform(t2)
newax.add_patch(polB)
I first tried to overlay the polygon on a line plotted from the x,y coordinates. However, the scales of the x and y axes were not equal (nor did I want them to be), so the polygon ended up looking all warped and stretched when rotated. I got around this by first adding a new axis with equal x/y scaling:
newax = fig.add_axes(ax.get_position(), frameon=False)
newax.set_xlim(-20,20)
newax.set_ylim(-20,20)
I could at least then rotate all I wanted and not have the warp issue. But then I needed to figure out how to basically connect the two axes so that I could plot the polygon on the new axis at a point referenced from the original axis. The way I figured to do this was by using transformations to go from the data coordinates on the original axis, converting them to display coordinates, and then inverting them back to data coordinates except this time at the data coordinates on the new axis:
inTrans = ax.transData.transform((x, y))
inv = newax.transData.inverted()
newTrans = inv.transform((inTrans[0], inTrans[1]))
newX = newTrans[0]
newY = newTrans[1]
It felt a little like some sort of Rube Goldberg machine to do it this way, but it did what I wanted.
In the end, I decided I didn't like this approach and went with keeping it simpler and using a fancy arrowhead instead of a polygon. Such is life...