I have a 2D coefficient array COEFF with size row x col and a position array POS with size n x 2.
The goal is to create a batched array BAT with size n x (2*l) x (2*l) where l is the half length of subarray.
It looks like this
BAT[i, :, :] = COEFF[POS[i, 1] - l:POS[i, 1] + l, POS[i, 0] - l:POS[i, 0] + l]
It is possible to generate BAT based on above sequential code. However, I'm wondering is there an efficient way to construct the BAT array in parallel.
Thanks!
I'm not aware of a perfectly satisfactory solution to mixing advanced indexing and slicing in that way. But the following may be acceptable (assuming that by "parallel" you mean "vectorised"):
import numpy as np
nrow, ncol = 7, 7
n, l = 3, 2
coeff = np.random.randint(0,10, (nrow,ncol))
pos = np.c_[np.random.randint(l, nrow-l+1, (n,)),np.random.randint(l, ncol-l+1, (n,))]
i = (pos[:, :1] + np.arange(-l, l))[:, :, None]
j = (pos[:, 1:] + np.arange(-l, l))[:, None, :]
print(coeff, '\n')
print(pos, '\n')
print(coeff[i, j])
Prints:
# [[7 6 7 6 3 9 9]
# [3 6 8 3 4 8 6]
# [3 7 4 7 4 6 8]
# [0 7 2 3 7 0 4]
# [8 5 2 0 0 1 7]
# [4 6 1 9 4 5 4]
# [1 6 8 3 4 5 0]]
# [[2 2]
# [3 2]
# [2 4]]
# [[[7 6 7 6]
# [3 6 8 3]
# [3 7 4 7]
# [0 7 2 3]]
# [[3 6 8 3]
# [3 7 4 7]
# [0 7 2 3]
# [8 5 2 0]]
# [[7 6 3 9]
# [8 3 4 8]
# [4 7 4 6]
# [2 3 7 0]]]
Related
Assume, I have a specific proportions of slots proportion = [30,30,20,10,10]and I want to feed it with 1 element and get it allocated one by one. For example, we start with [0,0,0,0,0] and add 1 we get [1,0,0,0,0]. What I have so far is that (based on this post answer):
def distribute_elements_in_slots(total, slots, pct):
distr = [total * pct[i] / 100 for i in range(slots)]
solid = [int(elem) for elem in distr]
short = [distr[i] - solid[i] for i in range(slots)]
leftover = int(round(sum(short)))
for i in range(leftover):
shortest = short.index(max(short))
solid[shortest] += 1
short[shortest] = 0
return solid
To feed 1 element at the time I've generated the list on 1's:
randomlist = []
for i in range(0,30):
n = random.randint(1,1)
randomlist.append(n)
print(randomlist)
And addition function to loop over that list:
x = 5
flexibility = [30, 30, 20, 10 ,10]
total = 0
cars = 0
for n in randomlist:
cars += 1
total += n
distributed = distribute_elements_in_slots(total, x, flexibility)
print(distributed)
But the broblem is this fucnction does not remeber the previous step.
1-[1, 0, 0, 0, 0]
2-[1, 1, 0, 0, 0]
3-[1, 1, 1, 0, 0]
4-[1, 1, 1, 1, 0] - on this step we have 4 elements in 4 slots.
5-[2, 2, 1, 0, 0] - on this step we took 1 from the fourth element and "gave" it to second.
But I want it it be like this:
1-[1, 0, 0, 0, 0]
2-[1, 1, 0, 0, 0]
3-[1, 1, 1, 0, 0]
4-[1, 1, 1, 1, 0]
5-[2, 1, 1, 1, 0]
This simple code gives a slot filling sequence without reallocations:
slots = 5
dist = np.array([0]*slots)
proportion = np.array([30,30,20,10,10])
for i in range(0,30):
total = max(dist.sum(),1)
prop = dist/total*100
error = proportion - prop
idx = np.argmax(error)
dist[idx] += 1
print(dist)
[1 0 0 0 0]
[1 1 0 0 0]
[1 1 1 0 0]
[1 1 1 1 0]
[1 1 1 1 1]
[2 1 1 1 1]
[2 2 1 1 1]
[2 2 2 1 1]
[3 2 2 1 1]
[3 3 2 1 1]
[4 3 2 1 1]
[4 4 2 1 1]
[4 4 3 1 1]
[4 4 3 2 1]
[4 4 3 2 2]
[5 4 3 2 2]
[5 5 3 2 2]
[5 5 4 2 2]
[6 5 4 2 2]
[6 6 4 2 2]
[7 6 4 2 2]
[7 7 4 2 2]
[7 7 5 2 2]
[7 7 5 3 2]
[7 7 5 3 3]
[8 7 5 3 3]
[8 8 5 3 3]
[8 8 6 3 3]
[9 8 6 3 3]
[9 9 6 3 3]
I'm looking to speed up my code that takes ~80 milliseconds for 300 sets to generate multiset_permutations from sympy. Ideally this would take only a few milliseconds; also the more items, the slower it gets.
What can I do to make my code faster? Multi-threading? Or convert to C? Any help here on speeding this up would be greatly appreciated.
import numpy as np
from time import monotonic
from sympy.utilities.iterables import multiset_permutations
milli_time = lambda: int(round(monotonic() * 1000))
start_time = milli_time()
num_indices = 5
num_items = 300
indices = np.array([list(multiset_permutations(list(range(num_indices)))) for _ in range(num_items)])
print(indices)
[[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
...
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]]
print('Multiset Perms:', milli_time() - start_time, 'milliseconds')
Multiset Perms: 88 milliseconds
** Code Update to Reduce extra computations by 2/3 **
import itertools
import numpy as np
from time import time, monotonic
from sympy.utilities.iterables import multiset_permutations
milli_time = lambda: int(round(monotonic() * 1000))
start_time = milli_time()
num_colors = 5
color_range = list(range(num_colors))
total_media = 300
def all_perms(elements):
if len(elements) <= 1:
yield elements # Only permutation possible = no permutation
else:
# Iteration over the first element in the result permutation:
for (index, first_elmt) in enumerate(elements):
other_elmts = elements[:index]+elements[index+1:]
for permutation in all_perms(other_elmts):
yield [first_elmt] + permutation
multiset = list(multiset_permutations(color_range))
# multiset = list(itertools.permutations(color_range))
# multiset = list(all_perms(color_range))
_range = range(total_media)
perm_indices = np.array([multiset for _ in _range])
print('Multiset Perms:', milli_time() - start_time)
Multiset Perms: 34 milliseconds
First of all, you do not need to recompute the permutations.
Moreover, np.array([multiset for _ in _range]) is expensive because Numpy have to transform multiset total_media times. You can solve that using np.array([multiset]).repeat(total_media, axis=0).
Finally, sympy is not the fastest implementation to perform such a computation. A faster implementation consists in using itertools instead:
num_colors = 5
total_media = 300
color_range = list(range(num_colors))
multiset = list(set(itertools.permutations(color_range)))
perm_indices = np.array([multiset], dtype=np.int32).repeat(total_media, axis=0)
However, this itertools-based implementation do not preserve the order of the permutations. If this is important, you can use np.sort on the Numpy array converted from multiset (with a specific axis and before applying repeat).
On my machine, this takes about 0.15 ms.
I have problem with execution of np.argpartition
I have nd.array
example = np.array([[5,6,7,3,4],[1,2,3,7,5],[6,7,4,2,3],[1,2,3,5,9],[2,3,6,1,2,]])
out: [[5 6 7 3 4]
[1 2 3 7 5]
[6 7 4 2 3]
[1 2 3 5 9]
[2 3 6 1 2]]
I can get indices for sorted array by np.argsort
print(np.argsort(example))
out:
[[3 4 0 1 2]
[0 1 2 4 3]
[3 4 2 0 1]
[0 1 2 3 4]
[3 0 4 1 2]]
I want to use np.argsort to economy some time for executing, because I need only 3 sorted element in each row of this array. I use this code to do it:
print(np.argpartition(example, 3, axis=1))
out: [[3 4 0 1 2]
[1 0 2 4 3]
[3 4 2 0 1]
[1 0 2 3 4]
[3 4 0 1 2]]
I expect that the first three indices of each row will match the indices in the sorted array, but this is not the caseŃ That doesn't work . I don't understand what I did wrong.
np.argpartition(example, k, axis=1) does not return sorted array for first k elements. It only returns indices such that only (k+1)th element is sorted. If you see in your output, only the 4th element matches with argsort()
If you want first three sorted elements, you have to give a list for k parameter
index_array = np.argpartition(example, [0,1,2], axis=1)
print(np.take_along_axis(example,index_array, axis=1)) ##this will give you first 3 sorted elements
I'm working on a way to find the lowest 1-Norm of a given Matrix using a permutation of its rows. The problem is that the permutation can't be fully random. There are 4 subsets of rows in the Matrix having a special parameter. I want to permute just the rows having this one parameter and keeping those on the same spot.
Ex. The first column defines the type of row.
A = [
1, val_11, val_12, ... #1. Row
2, val_21, val_22, ... #2. Row
2, val_31, val_32, ... #3. Row
2, val_41, val_42, ... #4. Row
1, val_51, val_52, ... #5. Row
]
So in this example I want to permute the 1. and 5. Row AND permute the 2., 3. and 4. Row keeping the Types like [1;2;2;2;1] in place.
You just have to carefully define your permutations. Fancy indexing will then do the job :
Example :
from numpy.random import randint
M0 = randint(10,size=(5,5))
after=[4,2,3,1,0]
M0 = M[after]
print(M0)
print(M)
[[4 9 3 0 0]
[3 1 7 6 0]
[6 6 5 0 9]
[0 4 7 1 3]
[0 0 1 0 6]]
[[0 0 1 0 6]
[6 6 5 0 9]
[0 4 7 1 3]
[3 1 7 6 0]
[4 9 3 0 0]]
I want to shuffle 3D matrix's rows but it doesn't work in a matrix
here is some example code
def shuffle(data,data_size):
for step in range(int(1*data_size)):
selected = int(np.random.uniform(0,data_size))
target = int(np.random.uniform(0,data_size))
print(data)
if selected!=target:
data[selected], data[target] = data[target], data[selected]
print(selected," and ",target, " are changed")
return data
data = [[[1,2,3,4],[1,2,3,5],[1,2,3,6]],
[[2,2,3,4],[2,2,3,5],[2,2,3,6]],
[[3,2,3,4],[3,2,3,5],[3,2,3,6]] ]
data = np.array(data)
data = shuffle(data,3)
in this code I want to shuffle data from some row list to another row list
but it's result doesn't work swaping but overwriting
here is result
[[[1 2 3 4]
[1 2 3 5]
[1 2 3 6]]
[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]
[[3 2 3 4]
[3 2 3 5]
[3 2 3 6]]]
2 and 1 are changed
[[[1 2 3 4]
[1 2 3 5]
[1 2 3 6]]
[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]
[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]]
1 and 0 are changed
[[[1 2 3 4]
[1 2 3 5]
[1 2 3 6]]
[[1 2 3 4]
[1 2 3 5]
[1 2 3 6]]
[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]]
0 and 2 are changed
[[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]
[[1 2 3 4]
[1 2 3 5]
[1 2 3 6]]
[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]]
2 and 1 are changed
how can i swap list in matrix?
thanks
import numpy as np
def shuffle(data,data_size):
for step in range(int(1*data_size)):
selected = int(np.random.uniform(0,data_size))
target = int(np.random.uniform(0,data_size))
print(data)
if selected!=target:
data[[selected, target]] = data[[target, selected]]
print(selected," and ",target, " are changed")
return data
data = [[[1,2,3,4],[1,2,3,5],[1,2,3,6]],
[[2,2,3,4],[2,2,3,5],[2,2,3,6]],
[[3,2,3,4],[3,2,3,5],[3,2,3,6]] ]
data = np.array(data)
data = shuffle(data,3)
If you want to shuffle along the first axis, just use np.random.shuffle:
data = np.array([
[[1,2,3,4],[1,2,3,5],[1,2,3,6]],
[[2,2,3,4],[2,2,3,5],[2,2,3,6]],
[[3,2,3,4],[3,2,3,5],[3,2,3,6]]
])
np.random.shuffle(data)
print(data)
Output:
[[[3 2 3 4]
[3 2 3 5]
[3 2 3 6]]
[[1 2 3 4]
[1 2 3 5]
[1 2 3 6]]
[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]]
If you want to shuffle along any other axis in data, you can shuffle the array view returned by np.swapaxes. For example, to shuffle the rows of the inner 2D matrices, do:
swap = np.swapaxes(data, 1, 0)
np.random.shuffle(swap)
print(data)
Output:
[[[1 2 3 6]
[1 2 3 4]
[1 2 3 5]]
[[2 2 3 6]
[2 2 3 4]
[2 2 3 5]]
[[3 2 3 6]
[3 2 3 4]
[3 2 3 5]]]