Develop Reference

How to vectorize this matrix building operation?

Problem:
Vectorize the building of a matrix where certain elements are a function determined by whether or not a particular triple index exists in a dictionary. Other elements are 0.
Code:
This is simple to implement using for loops:
# specify matrix size. In this example it is 4*4 but they get much, much bigger
mat_size = 4
# create empty numpy array
matt = np.zeros((mat_size, mat_size))
# index_list is a list of tuples which are subtracted element-wise in the for loops to form a new triple index used to access the coming dictionary
# the list of tuples gets larger as the matrix size increases
index_list = [(0, 0, 0), (0, 0, 1), (0, 1, 0), (0, 0, 2), (0, 1, 1), (0, 2, 0), (1, 1, 0)]
# the dictionary of functions:
# keys are a tuple of integers
# values are the functions to be evaluated to form the matrix
# there are more keys in the actual problem but this is for the minimal working example
func_dict = {(0, 0, 0): lambda a : a + 10,
(0, 0, -2): lambda a : a - 3,
(0, 0, -1): lambda a : a**5,
(0, 0, 1): lambda a : a*10 + 5,
(0, 1, 0): lambda a : a/3 + 2,
(0, 1, -2): lambda a : a*2 + a**4,
(0, 1, -1): lambda a : a/4 * 76,
(0, 1, 1): lambda a : a*19 / 3,
}
# only loop over half the matrix as they are always symmetric
for H in range(mat_size):
for J in range(H, mat_size):
x = index_list[J][0] - index_list[H][0] # subtract first tuple elements
y = index_list[J][1] - index_list[H][1] # subtract second tuple elements
z = index_list[J][2] - index_list[H][2] # subtract third tuple elements
# check if x,y,z is in the dictionary
if (x, y, z) in func_dict:
matt[H,J] = func_dict[x, y, z](3.43)
else:
matt[H,J] = 0
Output:
An upper triangular (with diagonal) matrix:
[[13.43 39.3 3.14333333 0. ]
[ 0. 13.43 65.17 39.3 ]
[ 0. 0. 13.43 0. ]
[ 0. 0. 0. 13.43 ]]
I parallelized the outer loop which is the current implementation I have been using and has been suitably fast enough. The matrix sizes are now becoming larger and after profiling this is the limiting step which I need to speed up as it gets called repeatedly.
I can feel this problem will allow for vectorization but after several attempts always results in a loop which then ends up being comparable with the parallelized version.
Is there an effective way to vectorize this problem, removing any loop(s)?

Fetch elements from a dictionary faster

I have a dictionary containing a mapping from color code to class index like the following:
color_to_class_idx = {(0, 0, 0) : 0, (180, 120, 120): 1, (80, 50, 50): 2, (140, 140, 140): 3, (4, 250, 7): 4, (150, 6, 51): 5, (0, 102, 200): 6, (233, 255, 7): 7, (255, 31, 0): 8, (120, 120, 120): 9}
Now, I have a list of color code values like the following:
list_ = [(0, 0, 0) , (80, 50, 50), (255, 255, 255)]
I would like to get another list with the class_idx. Note that, in the list_ there can be color codes for which are not present in the keys of color_to_class_idx. For these cases, it should have a default value (e.g. 0). So, the final output would look like - [0, 2, 0]
The list_ could be dimension 345600. So, speed matters for me. Following is the my implementation:
values = np.array([color_to_cls_idx.get(key, 0) for key in segmented_img_list])
But it is slow.
TIA

You can do a little bit better (by ~50%?) by converting most of your data structures to Numpy arrays. Here is an array of dictionary values, addressed by the color coordinates. Note that all elements that are not explicitly updated are 0:
lookup = np.zeros((256, 256, 256), dtype=int)
for i in color_to_class_idx:
lookup[i] = color_to_class_idx[i]
Here is the result array:
result = np.zeros(len(segmented_img_list))
And here is the lookup loop:
for i,key in enumerate(segmented_img_list):
result[i] = lookup[key]

select first nth list items within a for loop

What I have at the moment is the below loop which lights each led on a strand of 24 in turn.
while True:
for i in range(24):
pixels = [ (0,0,0) ] * numLEDs
pixels[i] = (100, 100, 100)
client.put_pixels(pixels)
time.sleep(0.02)
What I would like to have is that the previously lit leds stay on for each iteration. So the number of lit leds increases by one each time round.
I assumed I could simply select from the start of the list to the i'th item as below.
However this returns a "TypeError: 'int' object is not iterable".
I'm not really clear why this doesn't work.
while True:
for i in range(24):
pixels = [ (0,0,0) ] * numLEDs
pixels[:i] = (100, 100, 100)
client.put_pixels(pixels)
time.sleep(0.02)
While I've got your attention, is there a better way to time these loops other than using time.sleep()? Although I am using threading, the sleeps still cause some delays when the led patterns change.

The problem is that you are forever overwriting your current pixel state with all zeroes. If you define the pixel structure outside of your infinite while loop, and then adjust one at a time only, it should fix your problem. Try something like this:
numLEDs = 24
pixels = [ (0,0,0) ] * numLEDs
while True:
for i in range(numLEDs):
pixels[i] = (100, 100, 100)
client.put_pixels(pixels)
time.sleep(0.02)

In your first example, the loop turns off the LEDs by setting them to (0,0,0). Instead, why not use this:
while True:
for i in range(24):
pixels = [ (100,100,100) ] * (i) + [ (0,0,0) ] * (numLEDs - i)
client.put_pixels(pixels)
time.sleep(0.02)
This sets the first i elements of the list to (100,100,100) then the remaining to be (0,0,0).
If i = 5 and numLEDs = 15, you will get this output:
[(100, 100, 100), (100, 100, 100), (100, 100, 100), (100, 100, 100), (100, 100, 100), (0, 0, 0), (0, 0, 0), (0, 0, 0), (0, 0, 0), (0, 0, 0), (0, 0, 0), (0, 0, 0), (0, 0, 0), (0, 0, 0), (0, 0, 0)]
As you can see, this will leave the others on.

If you want to light ALL leds in a sequence, then turn them off at once etc etc etc ... etc
# constants
ON = (100, 100, 100)
OFF = (0, 0, 0)
N = 24
n = 0
while 1:
if n%N == 0:
px = [ OFF ] * N
client.put_pixels(px)
time.sleep(0.02)
px[n%N] = ON
client.put_pixels(px)
time.sleep(0.02)
n += 1

This is an answer dealing with the task of having two consecutive leds always turned on, so that if you mount your leds' array in the shape of a circle you see a double light moving on the circle with a frequency of approx 2 Hz.
I answered under a false impression of what the OP effectively asked, I have given another answer that addresses the REAL question but I'd like to leave here this answer 'cause it has its beauty, at least in the eyes of the beholder...
You want to compute, using simple module arithmetic, which LED must be turned OFF and which has to be turned ON, using the trick (that I hope it is allowed under your requirements) that we start with a partially lit board.
# constants
ON = (100, 100, 100)
OFF = (0, 0, 0)
N = 24
# initial state of the leds
px = [ OFF ] * N
client.put_pixels(px) # no leds on
# set initial conditions for the iteration
px[0] = ON
n = -1
# the never-ending loop
while True:
n +=1
i, j = (n-1)%N, (n+1)%N
px[i], px[j] = OFF, ON
client.put_pixels(px)
time.sleep(0.02)
Before the 1st iteration you have led 0 ON in the px list, after the 1st iteration you have 2 leds ON, in all further iterations you put one led OFF and turn another led ON, so in every moment you have 2 led ON.

Python how to loop through an array in numpy at C speed and store some positions

I am new to python, numpy and opencv. I am playing with the first example of harris corner detector from here. My objective is to get an ordered list of all the corners. With this simple code I am able to get the X and Y coordinates of the corners and their value:
height, width, depth = img.shape
print height, width
for i in range(0, height): #looping at python speed
for j in range(0, (width)):
if dst[i,j] > 0.9*dst.max():
print i, j, dst[i,j]
However, it is dead slow. I don't know how this is called but apparently with numpy one can loop through arrays at C speed and even assign values, example:
img[0:height, 0:width, 0:depth] = 0
Can I loop through an array and assign the position of interesting values in another variable? I.e. can I use this on my code to make it faster?

You can get a mask of elements that would pass the IF conditional statement. Next up, if you need the indices that would pass the condition, use np.where or np.argwhere on the mask. For the valid dst elements, index dst with the same mask, thus using boolean indexing. The implementation would look something like this -
mask = dst > 0.9*dst.max()
out = np.column_stack((np.argwhere(mask),dst[mask]))
If you would like to get those three printed outputs separately, you could do -
I,J = np.where(mask)
valid_dst = dst[mask]
Finally, if you would like to edit the 3D array img based on the 2D mask, you could do -
img[mask] = 0
This way, you would change the corresponding elements in img across all channels in one go.

First of all if you are using Python 2.X you should use xrange instead of range, this speeds things up. range in Python 3.X has the same implementation as xrange in Python 2.X.
If you want to iterate over numpy arrays why not use the numpy enumerator for iteration?
# creating a sample img array with size 2x2 and RGB values
# so the shape is (2, 2, 3)
> img = np.random.randint(0, 255, (2,2,3))
> img
> array([[[228, 61, 154],
[108, 25, 52]],
[[237, 207, 127],
[246, 223, 101]]])
# iterate over the values, getting key and value at the same time
# additionally, no nasty nested for loops
> for key, val in np.ndenumerate(img):
> print key, val
# this prints the following
> (0, 0, 0) 228
> (0, 0, 1) 61
> (0, 0, 2) 154
> (0, 1, 0) 108
> (0, 1, 1) 25
> (0, 1, 2) 52
> (1, 0, 0) 237
> (1, 0, 1) 207
> (1, 0, 2) 127
> (1, 1, 0) 246
> (1, 1, 1) 223
> (1, 1, 2) 101

Create new numpy array-scalar of flexible dtype

I have a working solution to my problem, but when trying different things I was astounded there wasn't a better solution that I could find. It all boils down to creating a single flexible dtype value for comparing and inserting into an array.
I have an RGB 24-bit image (so 8-bits for each R, G, and B) image array. It turns out for some actions it is best to use it as a 3D array with HxWx3 other times it is best to use it as a structured array with the dtype([('R',uint8),('G',uint8),('B',uint8)]). One example is when trying to relabel the image colors so that every unique color is given a different value. I do this with the following code:
# Given im as an array of HxWx3, dtype=uint8
from numpy import dtype, uint8, unique, insert, searchsorted
rgb_dtype = dtype([('R',uint8),('G',uint8),('B',uint8)]))
im = im.view(dtype=rgb_dtype).squeeze() # need squeeze to remove the third dim
values = unique(im)
if tuple(values[0]) != (0, 0, 0):
values = insert(values, 0, 0) # value 0 needs to always be (0, 0, 0)
labels = searchsorted(values, im)
This works beautifully, however I am tried to make the if statement look nicer and just couldn't find a way. So lets look at the comparison first:
>>> values[0]
(0, 0, 0)
>>> values[0] == 0
False
>>> values[0] == (0, 0, 0)
False
>>> values[0] == array([0, 0, 0])
False
>>> values[0] == array([uint8(0), uint8(0), uint8(0)]).view(dtype=rgb_dtype)[0]
True
>>> values[0] == zeros((), dtype=rgb_dtype)
True
But what if you wanted something besides (0, 0, 0) or (1, 1, 1) and something that was not ridiculous looking? It seems like there should be an easier way to construct this, like rgb_dtype.create((0,0,0)).
Next with the insert statement, you need to insert 0 for (0, 0, 0). For other values this really does not work, for example inserting (1, 2, 3) actually inserts (1, 1, 1), (2, 2, 2), (3, 3, 3).
So in the end, is there a nicer way? Thanks!

I could make insert() work for your case doing (note that instead of 0 it is used [0]):
values = insert(values, [0], (1,2,3))
giving (for example):
array([(0, 1, 3), (0, 0, 0), (0, 0, 4), ..., (255, 255, 251), (255, 255, 253), (255, 255, 255)],
dtype=[('R', 'u1'), ('G', 'u1'), ('B', 'u1')])
Regarding another way to do your if, you can do this:
str(values[0]) == str((0,0,0))
or, perhaps more robust:
eval(str(values[0])) == eval(str(0,0,0))