Assume, I have a specific proportions of slots proportion = [30,30,20,10,10]and I want to feed it with 1 element and get it allocated one by one. For example, we start with [0,0,0,0,0] and add 1 we get [1,0,0,0,0]. What I have so far is that (based on this post answer):
def distribute_elements_in_slots(total, slots, pct):
distr = [total * pct[i] / 100 for i in range(slots)]
solid = [int(elem) for elem in distr]
short = [distr[i] - solid[i] for i in range(slots)]
leftover = int(round(sum(short)))
for i in range(leftover):
shortest = short.index(max(short))
solid[shortest] += 1
short[shortest] = 0
return solid
To feed 1 element at the time I've generated the list on 1's:
randomlist = []
for i in range(0,30):
n = random.randint(1,1)
randomlist.append(n)
print(randomlist)
And addition function to loop over that list:
x = 5
flexibility = [30, 30, 20, 10 ,10]
total = 0
cars = 0
for n in randomlist:
cars += 1
total += n
distributed = distribute_elements_in_slots(total, x, flexibility)
print(distributed)
But the broblem is this fucnction does not remeber the previous step.
1-[1, 0, 0, 0, 0]
2-[1, 1, 0, 0, 0]
3-[1, 1, 1, 0, 0]
4-[1, 1, 1, 1, 0] - on this step we have 4 elements in 4 slots.
5-[2, 2, 1, 0, 0] - on this step we took 1 from the fourth element and "gave" it to second.
But I want it it be like this:
1-[1, 0, 0, 0, 0]
2-[1, 1, 0, 0, 0]
3-[1, 1, 1, 0, 0]
4-[1, 1, 1, 1, 0]
5-[2, 1, 1, 1, 0]
This simple code gives a slot filling sequence without reallocations:
slots = 5
dist = np.array([0]*slots)
proportion = np.array([30,30,20,10,10])
for i in range(0,30):
total = max(dist.sum(),1)
prop = dist/total*100
error = proportion - prop
idx = np.argmax(error)
dist[idx] += 1
print(dist)
[1 0 0 0 0]
[1 1 0 0 0]
[1 1 1 0 0]
[1 1 1 1 0]
[1 1 1 1 1]
[2 1 1 1 1]
[2 2 1 1 1]
[2 2 2 1 1]
[3 2 2 1 1]
[3 3 2 1 1]
[4 3 2 1 1]
[4 4 2 1 1]
[4 4 3 1 1]
[4 4 3 2 1]
[4 4 3 2 2]
[5 4 3 2 2]
[5 5 3 2 2]
[5 5 4 2 2]
[6 5 4 2 2]
[6 6 4 2 2]
[7 6 4 2 2]
[7 7 4 2 2]
[7 7 5 2 2]
[7 7 5 3 2]
[7 7 5 3 3]
[8 7 5 3 3]
[8 8 5 3 3]
[8 8 6 3 3]
[9 8 6 3 3]
[9 9 6 3 3]
Related
I'm looking to speed up my code that takes ~80 milliseconds for 300 sets to generate multiset_permutations from sympy. Ideally this would take only a few milliseconds; also the more items, the slower it gets.
What can I do to make my code faster? Multi-threading? Or convert to C? Any help here on speeding this up would be greatly appreciated.
import numpy as np
from time import monotonic
from sympy.utilities.iterables import multiset_permutations
milli_time = lambda: int(round(monotonic() * 1000))
start_time = milli_time()
num_indices = 5
num_items = 300
indices = np.array([list(multiset_permutations(list(range(num_indices)))) for _ in range(num_items)])
print(indices)
[[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
...
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]]
print('Multiset Perms:', milli_time() - start_time, 'milliseconds')
Multiset Perms: 88 milliseconds
** Code Update to Reduce extra computations by 2/3 **
import itertools
import numpy as np
from time import time, monotonic
from sympy.utilities.iterables import multiset_permutations
milli_time = lambda: int(round(monotonic() * 1000))
start_time = milli_time()
num_colors = 5
color_range = list(range(num_colors))
total_media = 300
def all_perms(elements):
if len(elements) <= 1:
yield elements # Only permutation possible = no permutation
else:
# Iteration over the first element in the result permutation:
for (index, first_elmt) in enumerate(elements):
other_elmts = elements[:index]+elements[index+1:]
for permutation in all_perms(other_elmts):
yield [first_elmt] + permutation
multiset = list(multiset_permutations(color_range))
# multiset = list(itertools.permutations(color_range))
# multiset = list(all_perms(color_range))
_range = range(total_media)
perm_indices = np.array([multiset for _ in _range])
print('Multiset Perms:', milli_time() - start_time)
Multiset Perms: 34 milliseconds
First of all, you do not need to recompute the permutations.
Moreover, np.array([multiset for _ in _range]) is expensive because Numpy have to transform multiset total_media times. You can solve that using np.array([multiset]).repeat(total_media, axis=0).
Finally, sympy is not the fastest implementation to perform such a computation. A faster implementation consists in using itertools instead:
num_colors = 5
total_media = 300
color_range = list(range(num_colors))
multiset = list(set(itertools.permutations(color_range)))
perm_indices = np.array([multiset], dtype=np.int32).repeat(total_media, axis=0)
However, this itertools-based implementation do not preserve the order of the permutations. If this is important, you can use np.sort on the Numpy array converted from multiset (with a specific axis and before applying repeat).
On my machine, this takes about 0.15 ms.
I want to shuffle 3D matrix's rows but it doesn't work in a matrix
here is some example code
def shuffle(data,data_size):
for step in range(int(1*data_size)):
selected = int(np.random.uniform(0,data_size))
target = int(np.random.uniform(0,data_size))
print(data)
if selected!=target:
data[selected], data[target] = data[target], data[selected]
print(selected," and ",target, " are changed")
return data
data = [[[1,2,3,4],[1,2,3,5],[1,2,3,6]],
[[2,2,3,4],[2,2,3,5],[2,2,3,6]],
[[3,2,3,4],[3,2,3,5],[3,2,3,6]] ]
data = np.array(data)
data = shuffle(data,3)
in this code I want to shuffle data from some row list to another row list
but it's result doesn't work swaping but overwriting
here is result
[[[1 2 3 4]
[1 2 3 5]
[1 2 3 6]]
[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]
[[3 2 3 4]
[3 2 3 5]
[3 2 3 6]]]
2 and 1 are changed
[[[1 2 3 4]
[1 2 3 5]
[1 2 3 6]]
[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]
[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]]
1 and 0 are changed
[[[1 2 3 4]
[1 2 3 5]
[1 2 3 6]]
[[1 2 3 4]
[1 2 3 5]
[1 2 3 6]]
[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]]
0 and 2 are changed
[[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]
[[1 2 3 4]
[1 2 3 5]
[1 2 3 6]]
[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]]
2 and 1 are changed
how can i swap list in matrix?
thanks
import numpy as np
def shuffle(data,data_size):
for step in range(int(1*data_size)):
selected = int(np.random.uniform(0,data_size))
target = int(np.random.uniform(0,data_size))
print(data)
if selected!=target:
data[[selected, target]] = data[[target, selected]]
print(selected," and ",target, " are changed")
return data
data = [[[1,2,3,4],[1,2,3,5],[1,2,3,6]],
[[2,2,3,4],[2,2,3,5],[2,2,3,6]],
[[3,2,3,4],[3,2,3,5],[3,2,3,6]] ]
data = np.array(data)
data = shuffle(data,3)
If you want to shuffle along the first axis, just use np.random.shuffle:
data = np.array([
[[1,2,3,4],[1,2,3,5],[1,2,3,6]],
[[2,2,3,4],[2,2,3,5],[2,2,3,6]],
[[3,2,3,4],[3,2,3,5],[3,2,3,6]]
])
np.random.shuffle(data)
print(data)
Output:
[[[3 2 3 4]
[3 2 3 5]
[3 2 3 6]]
[[1 2 3 4]
[1 2 3 5]
[1 2 3 6]]
[[2 2 3 4]
[2 2 3 5]
[2 2 3 6]]]
If you want to shuffle along any other axis in data, you can shuffle the array view returned by np.swapaxes. For example, to shuffle the rows of the inner 2D matrices, do:
swap = np.swapaxes(data, 1, 0)
np.random.shuffle(swap)
print(data)
Output:
[[[1 2 3 6]
[1 2 3 4]
[1 2 3 5]]
[[2 2 3 6]
[2 2 3 4]
[2 2 3 5]]
[[3 2 3 6]
[3 2 3 4]
[3 2 3 5]]]
I have array:
arr = np.array([1,2,3,2,3,4,3,2,1,2,3,1,2,3,2,2,3,4,2,1])
print (arr)
[1 2 3 2 3 4 3 2 1 2 3 1 2 3 2 2 3 4 2 1]
I would like find this pattern and return booelan mask:
pat = [1,2,3]
N = len(pat)
I use strides:
#https://stackoverflow.com/q/7100242/2901002
def rolling_window(a, window):
shape = a.shape[:-1] + (a.shape[-1] - window + 1, window)
strides = a.strides + (a.strides[-1],)
c = np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)
return c
print (rolling_window(arr, N))
[[1 2 3]
[2 3 2]
[3 2 3]
[2 3 4]
[3 4 3]
[4 3 2]
[3 2 1]
[2 1 2]
[1 2 3]
[2 3 1]
[3 1 2]
[1 2 3]
[2 3 2]
[3 2 2]
[2 2 3]
[2 3 4]
[3 4 2]
[4 2 1]]
I find positions of first values only:
b = np.all(rolling_window(arr, N) == pat, axis=1)
c = np.mgrid[0:len(b)][b]
print (c)
[ 0 8 11]
And positions another vals:
d = [i for x in c for i in range(x, x+N)]
print (d)
[0, 1, 2, 8, 9, 10, 11, 12, 13]
Last return mask by in1d:
e = np.in1d(np.arange(len(arr)), d)
print (e)
[ True True True False False False False False True True
True True True True False False False False False False]
Verify mask:
print (np.vstack((arr, e)))
[[1 2 3 2 3 4 3 2 1 2 3 1 2 3 2 2 3 4 2 1]
[1 1 1 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0]]
1 2 3 1 2 3 1 2 3
I think my solution is a bit over-complicated. Is there some better, more pythonic solution?
We can simplify things at the end with Scipy supported binary-dilation -
from scipy.ndimage.morphology import binary_dilation
m = (rolling_window(arr, len(pat)) == pat).all(1)
m_ext = np.r_[m,np.zeros(len(arr) - len(m), dtype=bool)]
out = binary_dilation(m_ext, structure=[1]*N, origin=-(N//2))
For performance, we can bring in OpenCV with its template matching capability, as we are basically doing the same here, like so -
import cv2
tol = 1e-5
pat_arr = np.asarray(pat, dtype='uint8')
m = (cv2.matchTemplate(arr.astype('uint8'),pat_arr,cv2.TM_SQDIFF) < tol).ravel()
Not sure how safe this is, but another method would be to read back to an as_strided view of the boolean output. As long as you only have one pat at a time it shouldn't be a problem I think, and it may work with more but I can't gurantee it because reading back to as_strided can be a bit unpredictable:
def vview(a): #based on #jaime's answer: https://stackoverflow.com/a/16973510/4427777
return np.ascontiguousarray(a).view(np.dtype((np.void, a.dtype.itemsize * a.shape[1])))
def roll_mask(arr, pat):
pat = np.atleast_2d(pat)
out = np.zeros_like(arr).astype(bool)
vout = rolling_window(out, pat.shape[-1])
vout[np.in1d(vview(rolling_window(arr, pat.shape[-1])), vview(pat))] = True
return out
np.where(roll_mask(arr, pat))
(array([ 0, 1, 2, 8, 9, 10, 11, 12, 13], dtype=int32),)
pat = np.array([[1, 2, 3], [3, 2, 3]])
print([i for i in arr[roll_mask(arr, pat)]])
[1, 2, 3, 2, 3, 1, 2, 3, 1, 2, 3]
It seems to work, but I wouldn't give this answer to a beginner!
I have a 2D coefficient array COEFF with size row x col and a position array POS with size n x 2.
The goal is to create a batched array BAT with size n x (2*l) x (2*l) where l is the half length of subarray.
It looks like this
BAT[i, :, :] = COEFF[POS[i, 1] - l:POS[i, 1] + l, POS[i, 0] - l:POS[i, 0] + l]
It is possible to generate BAT based on above sequential code. However, I'm wondering is there an efficient way to construct the BAT array in parallel.
Thanks!
I'm not aware of a perfectly satisfactory solution to mixing advanced indexing and slicing in that way. But the following may be acceptable (assuming that by "parallel" you mean "vectorised"):
import numpy as np
nrow, ncol = 7, 7
n, l = 3, 2
coeff = np.random.randint(0,10, (nrow,ncol))
pos = np.c_[np.random.randint(l, nrow-l+1, (n,)),np.random.randint(l, ncol-l+1, (n,))]
i = (pos[:, :1] + np.arange(-l, l))[:, :, None]
j = (pos[:, 1:] + np.arange(-l, l))[:, None, :]
print(coeff, '\n')
print(pos, '\n')
print(coeff[i, j])
Prints:
# [[7 6 7 6 3 9 9]
# [3 6 8 3 4 8 6]
# [3 7 4 7 4 6 8]
# [0 7 2 3 7 0 4]
# [8 5 2 0 0 1 7]
# [4 6 1 9 4 5 4]
# [1 6 8 3 4 5 0]]
# [[2 2]
# [3 2]
# [2 4]]
# [[[7 6 7 6]
# [3 6 8 3]
# [3 7 4 7]
# [0 7 2 3]]
# [[3 6 8 3]
# [3 7 4 7]
# [0 7 2 3]
# [8 5 2 0]]
# [[7 6 3 9]
# [8 3 4 8]
# [4 7 4 6]
# [2 3 7 0]]]
I have a problem with the instruction np.nonzero() in python. I want to take all the indices of a given list that are non zero. So, consider that I have the following code:
import numpy as np
from scipy.special import binom
M=4
N=3
def generate(N,nb):
states = np.zeros((int(binom(nb+N-1, nb)), N), dtype=int)
states[0, 0]=nb
ni = 0 # init
for i in xrange(1, states.shape[0]):
states[i,:N-1] = states[i-1, :N-1]
states[i,ni] -= 1
states[i,ni+1] += 1+states[i-1, N-1]
if ni >= N-2:
if np.any(states[i, :N-1]):
ni = np.nonzero(states[i, :N-1])[0][-1]
else:
ni += 1
return states
base = generate(M,N)
The result of base is given by:
base = [[3 0 0 0]
[2 1 0 0]
[2 0 1 0]
[2 0 0 1]
[1 2 0 0]
[1 1 1 0]
[1 1 0 1]
[1 0 2 0]
[1 0 1 1]
[1 0 0 2]
[0 3 0 0]
[0 2 1 0]
[0 2 0 1]
[0 1 2 0]
[0 1 1 1]
[0 1 0 2]
[0 0 3 0]
[0 0 2 1]
[0 0 1 2]
[0 0 0 3]]
The point is that for a given index j,k I want to take all the items in base that has non-zero components in the sites j,k, for example:
Taking j=0,k=1 I have to obtain:
result = [1 4 5 6]
which corresponds to the elements 1,4,5,6 of base that satisfies this condition. On the other hand, I have used the command:
np.nonzero((base[:, j]) & (base[:, k]))[0]
but it doesn't work correctly, any idea why?
First of all, the syntax for list index base[:, j] is wrong, use : [:][j] instead
also:
np.nonzero((base[:, j]) & (base[:, k]))[0]
won't work ,because the & sign is not applicable here..
you could use numpy like this:
b = np.array(base);
j=0;k=1;
np.nonzero(b.T[j]* b.T[k])[0]
which will give:
array([1, 4, 5, 6])