Develop Reference

Merge one tensor into other tensor on specific indexes in PyTorch

Any efficient way to merge one tensor to another in Pytorch, but on specific indexes.
Here is my full problem.
I have a list of indexes of a tensor in below code xy is the original tensor.
I need to preserve the rows (those rows who are in indexes list) of xy and apply some function on elements other than those indexes (For simplicity let say the function is 'multiply them with two),
xy = torch.rand(100,4)
indexes=[1,2,55,44,66,99,3,65,47,88,99,0]
Then merge them back into the original tensor.
This is what I have done so far:
I create a mask tensor
indexes=[1,2,55,44,66,99,3,65,47,88,99,0]
xy = torch.rand(100,4)
mask=[]
for i in range(0,xy.shape[0]):
if i in indexes:
mask.append(False)
else:
mask.append(True)
print(mask)
import numpy as np
target_mask = torch.from_numpy(np.array(mask, dtype=bool))
print(target_mask.sum()) #output is 89 as these are element other than preserved.
Apply the function on masked rows
zy = xy[target_mask]
print(zy)
zy=zy*2
print(zy)
Code above is working fine and posted here to clarify the problem
Now I want to merge tensor zy into xy on specified index saved in the list indexes.
Here is the pseudocode I made, as one can see it is too complex and need 3 for loops to complete the task. and it will be too much resources wastage.
# pseudocode
for masked_row in indexes:
for xy_rows_index in xy:
if xy_rows_index= masked_row
pass
else:
take zy tensor row and replace here #another loop to read zy.
But I am not sure what is an efficient way to merge them, as I don't want to use NumPy or for loop etc. It will make the process slow, as the original tensor is too big and I am going to use GPU.
Any efficient way in Pytorch for this?

Once you have your mask you can assign updated values in place.
zy = 2 * xy[target_mask]
xy[target_mask] = zy
As for acquiring the mask I don't see a problem necessarily with your approach, though using the built-in set operations would probably be more efficient. This also gives an index tensor instead of a mask, which, depending on the number of indices being updated, may be more efficient.
i = list(set(range(len(xy)))-set(indexes))
zy = 2 * xy[i]
xy[i] = zy
Edit:
To address the comment, specifically to find the complement of indices of i we can do
i_complement = list(set(range(len(xy)))-set(i))
However, assuming indexes contains only values between 0 and len(xy)-1 then we could equivalently use i_complement = len(set(indexes)), which just removes the repeated values in indexes.

How to modify rows of 2D numpy arrays individually

I have a dictionary of 2D arrays and I would like to normalize each row of each 2D array by its mean.
I have:
for key, value in sorted(baseline.items()):
for i in baseline[str(key)]:
i = i / np.mean(i)
Where:
baseline is a dict
baseline[str(key)] is a 2D numpy array
i is a 1D array
print(i) results in the appropriately updated values, however the individual rows across baseline.items() do not get updated.
What am I missing?

First of all, here is a solution:
for i in baseline.values():
i /= i.mean(axis=1, keepdims=True)
Now as to why. The loop for i in baseline[key]: binds a view into the row of a 2D array to the name i at each iteration. You don't need str(key) because the outer loop ensures that the keys are correct. In fact, avoid transforming the keys unnecessarily to avoid surprises, like if you accidentally get an integer key.
The line i = i / np.mean(i) does not do in-place division of the array by its mean. It computes the array i / np.mean(i), then rebinds the name i to the new array. The new array is then discarded on the next iteration.
You can fix this by re-assigning into the slice that i represents:
i[:] = i / np.mean(i)
Alternatively, you can perform the division in-place using the correct operator:
i /= np.mean(i)
As you can see in my solution, there is no need to iterate over the rows at all. np.mean is a vectorized function that can operate along any axis of an array. By setting keepdims=True, you ensure that the result has the right shape to be broadcasted right back over the original when you divide them.
A less flexible alternative to i.mean(axis=1, keepdims=True) specific for 2D arrays is
i.mean(axis=1)[:, None]

Numpy notation to replace an enumerate(zip(....))

I'm starting to use numpy. I get the slice notations and element-wise computations, but I can't understand this:
for i, (I,J) in enumerate(zip(data_list[0], data_list[1])):
joint_hist[int(np.floor(I/self.bin_size))][int(np.floor(J/self.bin_size))] += 1
Variables:
data_list contains two np.array().flatten() images (eventually more)
joint_hist[] is the joint histogram of those two images, it's displayed later with plt.imshow()
bin_size is the number of slots in the histogram
I can't understand why the coordinate in the final histogram is I,J. So it's not just that the value at a position in joint_hist[] is the result of some slicing/element-wise computation. I need to take the result of that computation and use THAT as the indices in joint_hist...
EDIT:
I indeed do not use the i in the loop actually - it's a leftover from previous iterations and I simply hadn't noticed I didn't need it anymore
I do want to remain in control of the bin sizes & the details of how this is done, so not particularly looking to use histogramm2D. I will later be using that for further image processing, so I'd rather have the flexibility to adapt my approach than have to figure out if/how to do particular things with built-in functions.

You can indeed gussy up that for loop using some numpy notation. Assuming you don't actually need i (since it isn't used anywhere):
for I,J in (data_list.T // self.bin_size).astype(int):
joint_hist[I, J] += 1
Explanation
data_list.T flips data_list on its side. Each row of data_list.T will contain the data for the pixels at a particular coordinate.
data_list.T // self.bin_size will produce the same result as np.floor(I/self.bin_size), only it will operate on all of the pixels at once, instead of one at a time.
.astype(int) does the same thing as int(...), but again operates on the entire array instead of a single element.
When you iterate over a 2D array with a for loop, the rows are returned one at a time. Thus, the for I,J in arr syntax will give you back one pair of pixels at a time, just like your zip statement did originally.
Alternative
You could also just use histogramdd to calculate joint_hist, in place of your for loop. For your application it would look like:
import numpy as np
joint_hist,edges = np.histogramdd(data_list.T)
This would have different bins than the ones you specified above, though (numpy would determine them automatically).

If I understand, your goal is to make an histogram or correlated values in your images? Well, to achieve the right bin index, the computation that you used is not valid. Instead of np.floor(I/self.bin_size), use np.floor(I/(I_max/bin_size)).astype(int). You want to divide I and J by their respective resolution. The result that you will get is a diagonal matrix for joint_hist if both data_list[0] and data_list[1] are the same flattened image.
So all put together:
I_max = data_list[0].max()+1
J_max = data_list[1].max()+1
joint_hist = np.zeros((I_max, J_max))
bin_size = 256
for i, (I, J) in enumerate(zip(data_list[0], data_list[1])):
joint_hist[np.floor(I / (I_max / bin_size)).astype(int), np.floor(J / (J_max / bin_size)).astype(int)] += 1

Can I call the index of an array?

So I have an image which I imported to python. The imread command basically gives me an array X,Y,Z where X and Y are the coordinates of the pixels and Z (which has four dimensions) gives me the RGB values at a given point (X,Y).
import matplotlib.image as img
import numpy as np
RawImg = img.imread('tek0000.bmp','RGB')
CrpImg = RawImg[14:208,12:256,:]
x_values = []
y_values = []
for row in CrpImg:
for cell in row:
print(np.nonzero)
if (cell == [136,136,0,255]).all:
My goal is to analyze the exact points in the array where the RGB configuration is [136,136,0,255]. These points are greenish-yellow. I want to add the X and Y values to lists or arrays so I can plot them.
In order to achieve this, I iterate over every point X and Y (row and column) of the array, and analyze the Z values. What I need is the coordinate (X,Y) of the cell in the for loop.
Basically, if the color in the point (X,Y) of the image is yellow, add that point (X,Y) to the list.
Surprisingly I cannot find pretty much anything online for what I think, is a relatively simple thing. I realize that I can interate using the following:
for i in range len(X axis) something like that, but I want to know if it is possible this way.

Not completely sure this is what you're looking for, but I think you want to get the index from inside the loop. The main ways to do this would be
loop using the index, e.g. for i in range(0,255): and then index into the array
iterate using enumerate, which returns an index as well as value in a collection
use the index method
I think the easiest option for you will be the index method.
for row in CrpImg:
for cell in row:
print(np.nonzero)
if (cell == [136,136,0,255]).all:
print(CrpImg.index(row), row.index(cell))
Note that this is going to give you the index inside your crop rather than the full image. You can either adjust (by adding 14 and 12), or you can iterate over the full image.

If you use enumerate from the standard library, you get access to a tuple containing a count and your values. The count starts at 0 by default
for row in CrpImg
becomes
for num, row in enumerate(CrpImg):
print(num)

Try using numpy.where:
indices = numpy.where(my_array == [136,136,0,255])

What is this bit of code doing?

for x, y in [np.int32(tr[-1]) for tr in self.tracks]:
cv2.circle(mask, (x, y), 5, 0, -1)
p = cv2.goodFeaturesToTrack(frame_gray, mask=mask, **feature_params)
if p is not None:
for x, y in np.float32(p).reshape(-1, 2):
self.tracks.append([(x, y)])
I am confused by these for loops, I am quite new to python 3.From what I gather it is first stepping through the x,y values for each tr (from the end? is that what the -1 indicates?)
anyway for each tr in its instance: self.tracks. The bottom for loop is doing the same in the numpy float array:p? Why are they using 'reshape'? What is that doing?
If someone wouldn't mind perhaps just stepping through it for me
I'd greatly appreciate.

self.tracks seems to be an (n,2) list. The outer loop takes each of these values, turns them two by two into integers x and y, then performs a function/method cv2.circle with an object mask and several other parameters. container[-1] indicates that you want the value of the last index of container.
The values of the function/method goodFeaturesToTrack are assigned to p(which seem to be an array or None). ** indicates that feature_params is a dictionary of parameters(if a function is defined as myfunc(a,b=2,c=3,d=5) you could change some of these values for that
function call by calling myfunc("value of a",**mydict) with mydict being a dictionary containing zero or more of the optional variables a, b and c (eg mydict={'b':8,d:0} would change b and d from their default values to 8 and 0 respectively.
a new (float-valued) x and y are then extracted from a reshape of p and appended back to the list self.tracks as a pair.
The -1 in the reshape indicate that you dont care about how long the given axis as long as the other axis has the right shape. eg. an array of 10 values would be reshaped to (5,2) a (4,4) would be reshaped to (8,2) etc. This could have been found by searching for numpy.reshape:
newshape : int or tuple of ints
The new shape should be compatible with the original shape. If an >integer, then the result will be a 1-D array of that length. One shape >dimension can be -1. In this case, the value is inferred from the length of >the array and remaining dimensions.