I am having some problems with Affine tranformation coefficients while creating a new GeoTIFF file. What I am doing is ETL on a scientific dataset that results in a 2D Ndarray along with a set of meshgrid Ndarrays that contain Lat and Lon. Both the meshgrids and the dataset arrays have the same dimensions of 645 x 980. From what I understand the GeoTIFF requires a list of Affine coefficients when created from Python GDAL via the SetGeoTransform() method. The list has the form of [xllcorner, xrotation, x_cellsize, yllcorner, yrotation, y_cellsize]. My approach to this is similar to what is outlined here: http://adventuresindevelopment.blogspot.com/2008/12/python-gdal-adding-geotiff-meta-data.html
At this point is where I run into problems. I calculate the xllcorner and the yllcorner using the min() method for the two meshgrid arrays for lat & lon respectively, and I manually calculate the x and y cellsize by applying the formula [max-min]/dimension size with the x dimension size being the x axis size for the lons meshgrid and the y dimension size being the y axis size for the lats meshgrid. When I apply this method and try to write out the array band via GetRasterBand().WriteArray() I get this error message:
Traceback (most recent call last):
...
raise ValueError("array larger than output file, or offset off edge")
ValueError: array larger than output file, or offset off edge
Therefore I assume that I have composed my affine coefficients incorrectly but given the data this makes no sense to me. I even made sure that the Spatial Reference System was set to WGS:84 before attempting the affine coefficient creation. So my question is how to properly create the Affine coefficients with lat/lon meshgrids and a data array that share common dimensions? I think my cell size calculation can't simply be lat/lon differences; but I am not sure.
This error is typically shown when the expected array shape does not match. For instance, see what shape the expected shape is with:
band = src.GetRasterBand(1)
arr = band.ReadAsArray()
print(arr.shape) # (656L, 515L)
This will need to be the shape of the numpy array to be written:
assert other_array.shape == arr.shape
band.WriteArray(other_array)
And to raise the same ValueError, change the shape so it is longer in one dimension, e.g.:
band.WriteArray(other_array.T)
As for affine transformations, this is probably not raising any errors, as it is often just stored as data. GIS rasters typically register the world coordinate in the upper-left corner, and use a -dy value to count rows downwards. However, using a lower-left corner with +dy is usually fine by most software. It will just be upside down when comparing the array as a printed matrix versus mapped raster.
Related
I am trying to write a function that extracts a 2D slice in a non-orthogonal plane from a 3D volume using numpy. The non-orthogonal slice obtained should be a rectangular two-dimensional array of shape (n, m), while the input volume should be a three-dimensional numpy array of shape (i, j, k).
So far I have tried to create a function that receives the volume, the plane normal and a point that belongs to the plane as inputs. I'm representing the plane normal and the point with numpy arrays of shape (3,). I am quite certain the function should follow these steps:
The function should first create a grid with the indices of the volume coordinates.
The function should define the plane using the dot product of the normal and the point.
The function should find the coordinates of the bounding box that contains the entire image slice. It is important to note that, except for specific edge cases where one of the normal coefficients is 0, most bounding boxes should end up with its corners having a variable amount of coordinates from outside the image.
The function should interpolate the slice points from the bounding box using the volume, as some of the coordinates contained by the slice may not be integers and thus will not actually exist in the image.
The non-orthogonal slice obtained from interpolation should then be returned.
I am stuck at step 3. I have gotten steps 4 and 5 to work using orthogonal planes, which are easy to obtain using numpy slicing, but I have been stuck for days trying to get my head around how to find the bounding box coordinates (even though I do not think this should be a hard problem). Any help would be greatly welcome!
I am handling a set of data recorded by a 2D detector. Therefore, the data are represented by three arrays: x and y labelling the coordinate of a pixel and intensity storing the measured signal.
For example, a 6x6 grid will give a set of data:
xraw = np.array([0,1,2,3,4,5,0,1,2,3,4,5,...])
yraw = np.array([0,0,0,0,0,0,1,1,1,1,1,1,...])
intensity = np.array([i_00,i_01,i_02,i_03,i_04,i_05,i_10,i_11,...])
Due to various reasons, such as pixel defects, some of the data points are discarded in the raw data. Therefore, xraw, yraw, intensity have a size smaller than 36 (if that's a 6x6 grid), with, say, the point at (2,3) missing.
The intensity data needs further treatment by an element-wise multiplication with another array. This treatment array is from theoretical calculation and so it has a size of nxn (6x6 in this case). However, as some of the points in the true data are missing, the two arrays have different sizes.
I can use a loop to check for the missing points and eliminate the corresponding element in the treatment array. I wonder if there are some methods in numpy that take care of such operations. Thanks
First, construct the indices of available and all possible pixel positions by
avail_ind = yraw * h + xraw
all_ind = np.arange(0, h * w)
where h and w is the image's height and width in pixels.
Then, find the indices of the missing pixels by
missing_ind = all_ind[~np.in1d(all_ind, avail_ind)]
Once having the missing indices, use np.delete to construct a copy of the treatment_array with elements at the indices removed, then simply multiply that with your intensity array.
result = intensity * np.delete(treatment_array, missing_ind)
Case description
I have a set of spectral maps (intensity dependent on time and frequency) for a set of detectors which can be fit into a 3D array BlockDataset of size M x N x K (here: M = number of frequencies, N number of time steps and K is the number of detectors).
The M frequencies are log-spaced and the K detectors are normally indexed by a tuple consiting of 2 angles but for brevity I'm considering only one angle. The N time values are equidistant.
Creating a HoloViews dataset from BlockDataset with appropriate value arrays for all of the dimensions is possible, but requires me to switch from a simple hv.Image display to a hv.QuadMesh display.
Problem description
If the dataset is created with actual angle values, instead of just detector numbers, a conversion to a HoloMap fails with the following error:
DataError: The shape of the intensity value array does not match the expected dimensionality indicated by the key dimensions. Expected 2-D array, found 3-D array.
If detector numbers (integers) are used instead of angles (floating point numbers) there's no problem.
Code
timeDim = hv.Dimension("time", label="Time", unit="sec", values=times)
freqDim = hv.Dimension("frequency", label = "Angular frequency", unit="$\\frac{rad}{s}", values=omega)
polarAngleDim = hv.Dimension("angle", label=" $\varphi$", unit="rad", values=angles[:,0])
intensityDim = hv.Dimension("intensity", label="Intensity $\\frac{d^2 W}{d\Omega d\omega}(t,\vartheta,\varphi)", unit="J/(s srad)")
hvDatasetNatural = hv.Dataset((times, angles[:,0], omega, BlockDataset.transpose()), [timeDim, polarAngleDim, freqDim], intensityDim)
subset = hvDatasetNatural.select( angle=list(angles[selectedIndices,0]) )
img = subset.to( new_type=hv.QuadMesh, kdims=[timeDim, freqDim])
The selection of a subset appears to work properly, but neither the conversion of the subset, nor of the entire dataset to QuadMesh works.
Note again: times are lin-spaced float values, angles are nonlinearly spaced floats and omega are log-spaced float values.
Query
What may be the problem here? I.e., why doesn't .to() work on the dataset when 2 of the 3 dimensions are non-equidistant, non-integer values but it works well if only omega is kept non-equidistant?
I can construct a QuadMesh for a specific angle using hv.QuadMesh( (...), kdim=[..]) and hence essentially unwrapping the original object.
(an extra) Why does an aggregation along the, e.g., time dimension using subset.reduce(timeDim, np.sum) work, but subset.reduce(timeDim, np.trapz) fails with:
DataError: None of the available storage backends were able to support the supplied data format. GridInterface raised following error:
GridInterface interface requires at least one value dimension.
Recently i was struggling trying to take the pixel values of a 3D volume (np array) using specific space coordinate of a STL object.
The STL object is spatially overlapped with the 3D volume but the latter has no coordinate and so i don't know how to pick pixel values corresponding to the STL coordinates.
Any idea?
If the STL object is truly in the 3d volume's coordinate space, then you can simply STL's coordinate as an index to lookup the value from the 3d array. This lookup does nearest neighbor interpolation of the 3d image. For better looking results you'd want to do linear (or even cubic) interpolation of the nearby pixels.
In most 3d imaging tasks, those coordinate spaces do not align. So there is a transform to go from world space to 3d volume space. But if all you have is a 3d numpy array, then there is no transformation information.
Update:
To index into the 3d volume take the X, Y, Z coordinates of your point from the STL object and convert them into integer value I, J, K. Then lookup in the numpy array using I,J,K as indices: np_array[K][J][I]. I think you have to reverse the order of the indices because of the array ordering numpy uses.
When you way 3d array and the STL align in python, how are you showing that? The original DICOM or Nifti certainly have world coordinate transformations in the metadata.
I have a cube of particles which I've projected onto a 2D grid, Projecting the particles onto the grid by a clouds in cells and weighting them by a scalar.
I would then like the gradient of the scalar at every grid point. In 2D I am doing this using np.gradient and I get two arrays with the gradient in the x and y directions:
gradx, grady = np.gradient(grid)
Does anyone have any idea how I can generalize this to 3 Dimensions? The Clouds in Cells in 3D is fine but I am then left with a grid with the shape (700, 700, 700).
As far as I can see np.gradient can't deal with this?
Thanks,
Daniel
The Numpy documentation indicates that gradient works for any dimensions:
numpy.gradient(f, *varargs)
Return the gradient of an N-dimensional array.
The gradient is computed using central differences in the interior and
first differences at the boundaries. The returned gradient hence has
the same shape as the input array.
Parameters :
f: array_like. An N-dimensional array containing samples
of a scalar function.
*varargs: 0, 1, or N scalars specifying the sample distances in each direction, that is: dx, dy, dz, ... The default distance is 1.
Returns :
g: ndarray. N arrays of
the same shape as f giving the derivative of f with respect to each
dimension.
Seems like you should be able to extend your 2-dimensional code to 3D like you would expect.