Improving performance of numpy mapping operation - python

I have a numpy array of size (4, X, Y), where the first dimension stands for an (R,G,B,A) quadruplet.
My aim is to transpose each X*Y RGBA quadruplets to X*Y floating-point values, given a dictionary matching them.
My current code is as follows:
codeTable = {
(255, 255, 255, 127): 5.5,
(128, 128, 128, 255): 6.5,
(0 , 0 , 0 , 0 ): 7.5,
}
for i in range(0, rows):
for j in range(0, cols):
new_data[i,j] = codeTable.get(tuple(data[:,i,j]), -9999)
Where data is a numpy array of size (4, rows, cols), and new_data is of size (rows, cols).
The code is working fine, but takes quite a long time. How should I optimize that piece of code?
Here is a full example:
import numpy
codeTable = {
(253, 254, 255, 127): 5.5,
(128, 129, 130, 255): 6.5,
(0 , 0 , 0 , 0 ): 7.5,
}
# test data
rows = 2
cols = 2
data = numpy.array([
[[253, 0], [128, 0], [128, 0]],
[[254, 0], [129, 144], [129, 0]],
[[255, 0], [130, 243], [130, 5]],
[[127, 0], [255, 120], [255, 5]],
])
new_data = numpy.zeros((rows,cols), numpy.float32)
for i in range(0, rows):
for j in range(0, cols):
new_data[i,j] = codeTable.get(tuple(data[:,i,j]), -9999)
# expected result for `new_data`:
# array([[ 5.50000000e+00, 7.50000000e+00],
# [ 6.50000000e+00, -9.99900000e+03],
# [ 6.50000000e+00, -9.99900000e+03], dtype=float32)

Here's an approach that returns your expected result, but with such a small amount of data it's hard to know if this will be faster for you. Since I've avoided the double for loop, however, I imagine you'll see a pretty decent speedup.
import numpy
import pandas as pd
codeTable = {
(253, 254, 255, 127): 5.5,
(128, 129, 130, 255): 6.5,
(0 , 0 , 0 , 0 ): 7.5,
}
# test data
rows = 3
cols = 2
data = numpy.array([
[[253, 0], [128, 0], [128, 0]],
[[254, 0], [129, 144], [129, 0]],
[[255, 0], [130, 243], [130, 5]],
[[127, 0], [255, 120], [255, 5]],
])
new_data = numpy.zeros((rows,cols), numpy.float32)
for i in range(0, rows):
for j in range(0, cols):
new_data[i,j] = codeTable.get(tuple(data[:,i,j]), -9999)
def create_output(data):
# Reshape your two data sources to be a bit more sane
reshaped_data = data.reshape((4, -1))
df = pd.DataFrame(reshaped_data).T
reshaped_codeTable = []
for key in codeTable.keys():
reshaped = list(key) + [codeTable[key]]
reshaped_codeTable.append(reshaped)
ct = pd.DataFrame(reshaped_codeTable)
# Merge on the data, replace missing merges with -9999
result = df.merge(ct, how='left')
newest_data = result[4].fillna(-9999)
# Reshape
output = newest_data.reshape(rows, cols)
return output
output = create_output(data)
print(output)
# array([[ 5.50000000e+00, 7.50000000e+00],
# [ 6.50000000e+00, -9.99900000e+03],
# [ 6.50000000e+00, -9.99900000e+03])
print(numpy.array_equal(new_data, output))
# True

The numpy_indexed package (disclaimer: I am its author) contains a vectorized nd-array capable variant of list.index, which can be used to solve your problem efficiently and concisely:
import numpy_indexed as npi
map_keys = np.array(list(codeTable.keys()))
map_values = np.array(list(codeTable.values()))
indices = npi.indices(map_keys, data.reshape(4, -1).T, missing='mask')
remapped = np.where(indices.mask, -9999, map_values[indices.data]).reshape(data.shape[1:])

Related

Bubble up non null rows of a 3-D Tensor PyTorch

I am having some problem trying to convert the following function into a function only manipulating tensors.
def valid_sequence_output(sequence_output, valid_mask):
bs, max_len, feat_dim = sequence_output.shape
valid_output = torch.zeros(bs, max_len, feat_dim, dtype=torch.float32)
for i in range(bs):
jj = -1
for j in range(max_len):
if valid_mask[i][j].item() == 1:
jj += 1
valid_output[i][jj] = sequence_output[i][j]
return valid_output
where the input tensors can be created as follow:
size = ((2,5,2))
sequence_output = torch.randint(0, 250, size=size)
valid_mask = torch.randint(0, 2, size=size[:2])
I basically aim at “bubbling up” the non null rows of sequence_output. As an example having sequence_output equal to:
tensor([[[ 0, 0],
[ 15, 47],
[124, 230],
[ 0, 0],
[ 65, 31]],
[[ 0, 0],
[ 0, 0],
[ 0, 0],
[139, 228],
[224, 205]]])
I am trying to obtain the following tensor:
tensor([[[ 15, 47],
[124, 230],
[ 65, 31],
[ 0, 0],
[ 0, 0]],
[[139, 228],
[224, 205],
[ 0, 0],
[ 0, 0],
[ 0, 0]]])
If somebody has a suggestion on how to do this, I would really appreciate :D
I managed to come up with a quite nasty solution (which he's probably suboptimal) by constructing a matrix A of zeros and ones that will swap rows of X when performing a matrix multiplication...
def vso(seq_out, valid_mask):
X = torch.where(valid_mask.unsqueeze(-1) == 1, seq_out, torch.zeros_like(seq_out))
bs, max_len, _ = X.shape
tu = torch.unique(torch.nonzero(X)[:, :2], dim=0)
batch_axis = tu[:, 0]
rows_axis = tu[:, 1]
a = torch.arange(bs).repeat(batch_axis.shape).reshape(batch_axis.shape[0], -1).T
T = torch.cumsum(batch_axis == a, dim=1) - 1
cols_axis = T[batch_axis, torch.arange(batch_axis.shape[0])]
A = torch.zeros((bs, max_len, max_len))
A[(batch_axis, cols_axis, rows_axis)] = 1
valid_output = torch.matmul(A, X)
return valid_output
Still looking for better answers though !

Numpy: How to replace array with single value with vectorization?

I have an image saved as numpy array of shape [Height, Width, 3] and I want to replace every pixel with another value based on the color of pixel, so the final array will have a shape [Height, Weight].
My solution with for loop works but it's pretty slow. How can I use Numpy vectorization to make it more efficient?
image = cv2.imread("myimage.png")
result = np.zeros(shape=(image.shape[0], image.shape[1],))
for h in range(0, result.shape[0]):
for w in range(0, result.shape[1]):
result[h, w] = get_new_value(image[h, w])
Here is get_new_value function:
def get_new_value(array: np.ndarray) -> int:
mapping = {
(0, 0, 0): 0,
(0, 0, 255): 5,
(0, 100, 200): 8,
# ...
}
return mapping[tuple(array)]
you can use np.select() as shown below:
img=np.array(
[[[123 123 123]
[130 130 130]]
[[129 128 128]
[162 162 162]]])
condlist = [img==[123,123,123], img==[130, 130, 130], img==[129, 129, 129], img==[162, 162, 162]]
choicelist = [0, 5, 8, 9]
img_replaced = np.select(condlist, choicelist)
final = img_replaced[:, :, 0]
print('img_replaced')
print(img_replaced)
print('final')
print(final)
condlist is your list of colour values and choicelist is the list of replacements.
np.select then returns three channels and you just need to take one channel from that to give the array 'final' which is the format you want I believe
output is:
img_replaced
[[[0 0 0]
[5 5 5]]
[[0 0 0]
[9 9 9]]]
final
[[0 5]
[0 9]]
so code specific to your example and shown colour mappings would be:
image = cv2.imread("myimage.png")
condlist = [image==[0, 0, 0], image==[0, 0, 255], image==[0, 100, 200]]
choicelist = [0, 5, 8]
img_replaced = np.select(condlist, choicelist)
result = img_replaced[:, :, 0]

Convert multi-dimensional Numpy array to 2-dimensional array based on color values

I have an image which is read as a uint8 array with the shape (512,512,3).
Now I would like to convert this array to a uint8 array of shape (512,512,1), where each pixel value in the third axis are converted from a color value [255,0,0] to a single class label value [3], based on the following color/class encoding:
1 : [0, 0, 0],
2 : [0, 0, 255],
3 : [255, 0, 0],
4 : [150, 30, 150],
5 : [255, 65, 255],
6 : [150, 80, 0],
7 : [170, 120, 65],
8 : [125, 125, 125],
9 : [255, 255, 0],
10 : [0, 255, 255],
11 : [255, 150, 0],
12 : [255, 225, 120],
13 : [255, 125, 125],
14 : [200, 100, 100],
15 : [0, 255, 0],
16 : [0, 150, 80],
17 : [215, 175, 125],
18 : [220, 180, 210],
19 : [125, 125, 255]
What is the most efficient way to do this? I thought of looping through all classes and using numpy.where, but this is obviously time-consuming.
You could use giant lookup table. Let cls be [[0,0,0], [0,0,255], ...] of dtype=np.uint8.
LUT = np.zeros(size=(256,256,256), dtype='u1')
LUT[cls[:,0],cls[:,1],cls[:,2]] = np.arange(cls.shape[1])+1
img_as_cls = LUT[img[...,0],img[...,1], img[...,2]]
This solution is O(1) per pixel. It is also quite cache efficient because a small part of entries in LUT are actually used. It takes circa 10ms to process 1000x1000 image on my machine.
The solution can be slightly improved by converting 3-color channels to 24-bit integers.
Here is the code
def scalarize(x):
# compute x[...,2]*65536+x[...,1]*256+x[...,0] in efficient way
y = x[...,2].astype('u4')
y <<= 8
y +=x[...,1]
y <<= 8
y += x[...,0]
return y
LUT = np.zeros(2**24, dtype='u1')
LUT[scalarize(cls)] = 1 + np.arange(cls.shape[0])
simg = scalarize(img)
img_to_cls = LUT[simg]
After optimization it takes about 5ms to process 1000x1000 image.
One way: separately create the boolean arrays with True values where the input's pixel value matches one of the palette values, and then use arithmetic to combine them. Thus:
palette = [
[0, 0, 0],
[0, 0, 255],
[255, 0, 0],
# etc.
]
def palettized(data, palette):
# Initialize result array
shape = list(data.shape)
shape[-1] = 1
result = np.zeros(shape)
# Loop and add each palette index component.
for value, colour in enumerate(palette, 1):
result += (data == colour).all(axis=2) * value
return result
Here's one based on views -
# https://stackoverflow.com/a/45313353/ #Divakar
def view1D(a, b): # a, b are arrays
# This function gets 1D view into 2D input arrays
a = np.ascontiguousarray(a)
b = np.ascontiguousarray(b)
void_dt = np.dtype((np.void, a.dtype.itemsize * a.shape[-1]))
return a.view(void_dt).ravel(), b.view(void_dt).ravel()
def img2label(a, maps):
# Get one-dimension reduced view into input image and map arrays.
# We need to reshape image to 2D, then feed it to view1D to get 1D
# outputs and then reshape 1D image to 2D
A,B = view1D(a.reshape(-1,a.shape[-1]),maps)
A = A.reshape(a.shape[:2])
# Trace back positions of A in B using searchsorted. This gives us
# original order, which is the final output.
sidx = B.argsort()
return sidx[np.searchsorted(B,A,sorter=sidx)]
Given that your labels start from 1, you might want to add 1 to the output.
Sample run -
In [100]: # Mapping array
...: maps = np.array([[0, 0, 0],[0, 0, 255],\
...: [255, 0, 0],[150, 30, 150]],dtype=np.uint8)
...:
...: # Setup random image array
...: idx = np.array([[0,2,1,3],[1,3,2,0]])
...: img = maps[idx]
In [101]: img2label(img, maps) # should retrieve back idx
Out[101]:
array([[0, 2, 1, 3],
[1, 3, 2, 0]])

How to use numpy.where to change all pixels of an image?

I have an image of shape (300,300,3) consisting of these pixels [255, 194, 7],[224, 255, 8],[230, 230, 230],[11, 102, 255]. I want to change this pixel [230, 230, 230] to [255,255,255]. And rest other pixels to [0,0,0]. So I'm applying numpy where function to switch the pixels. Below is the code:
import numpy
im = numpy.array([[[255, 194, 7],[224, 255, 8],[230, 230, 230],[11, 102, 255]]])
im[np.where((im == [230, 230, 230]).all(axis = 2))] = [255,255,255]
im[np.where((im != [255,255,255]).all(axis = 2))] = [0,0,0]
The first code is working fine, but all the pixels that have 255 in it like [11, 102, 255] doesnot get flipped at all in the second line. and the image remains same. Can anyone tell me what I'm doing wrong ?
import numpy as np
im = np.array([[[255, 194, 7],[224, 255, 8],[230, 230, 230],[11, 102, 255]]])
Like this?
Make a mask and use it to change the values.
>>> mask = im == 230
>>> im[mask] = 255
>>> im[np.logical_not(mask)] = 0
>>> im
=> array([[[ 0, 0, 0],
[ 0, 0, 0],
[255, 255, 255],
[ 0, 0, 0]]])
Or using numpy.where
>>> np.where(im==230, 255, 0)
=> array([[[ 0, 0, 0],
[ 0, 0, 0],
[255, 255, 255],
[ 0, 0, 0]]])
try
np.array_equal(arr1, arr2)

Numpy: vectorized operations to create a 3D array

I am learning Python and would like to find an efficient way to solve this problem using Numpy.
I currently have a 4x8 array containing random integers:
import numpy as np
n = 3
k = np.random.randint(n, size = (4,8))
Each number represents a color defined by its RGB value in a nx3 array:
colors = np.array([[0 , 0 , 0 ],
[0 , 100, 255],
[255, 100, 0 ]])
I would like to use these numbers to create a new 4x8x3 array where the first two dimensions represent pixels locations, and the third dimension the color of each pixel. This could be thought of as number painting. For example, if k[3,4] = 2, then myArray[3,4,:] = [255 100 0].
I am getting familiar with Numpy tools, but I am unsure of what I should be looking for exactly. Since the array k will eventually be much larger (I'm thinking ~640x480) and contain more than n = 3 non-random colors, I would like to use vectorized operations in order to speed up the process (and learn a bit more about them). Is this the most efficient way to do it?
IIUC, all you need to do is index into colors with k:
>>> k = np.random.randint(n, size = (2,4))
>>> out = colors[k]
>>> out
array([[[ 0, 100, 255],
[255, 100, 0],
[255, 100, 0],
[255, 100, 0]],
[[ 0, 100, 255],
[ 0, 100, 255],
[255, 100, 0],
[255, 100, 0]]])
>>> out.shape
(2, 4, 3)
>>> all((out[i]==colors[c]).all() for i,c in np.ndenumerate(k))
True

Categories

Resources