Develop Reference

Removing indexes to match array dimensions

I have two arrays (x, y) with different values and I am trying to find the median for y for values in x < 100. My problem is that I have filtered out some values in array x so the arrays are not the same shape. Is there a way I can remove the those indexes that I removed in array y in array x?
For example that they both are 24, 36 but after the filtering array y is 22, 32 and x is still 24, 36. How can I remove the same indexes? lets say I removed index 4, 7 and 9, 14. How can I remove those exact same ones in array x?
My code if needed. data_mg is y and data_dg is x.
data_mg = image_data_mg[0].data[0:x, 0:y].astype('float')
data_err = image_data_err[0].data[0:x, 0:y].astype('float')
data_dg = image_data_dg[0].data[0:x, 0:y].astype('float')
data_mg[data_mg == 0] = np.nan
data_err[data_err == 0] = np.nan
data_dg[data_dg == 0] = np.nan
data_mg = data_mg[data_mg/data_err > 2]
data_dg = np.ndarray.flatten(data_dg)
data_dg = data_dg[data_mg]
data_mg = np.ndarray.flatten(data_mg)
data_mg = data_mg[np.logical_not(np.isnan(data_mg))]
data_dg = np.ndarray.flatten(data_dg)
data_dg = data_dg[np.logical_not(np.isnan(data_dg))]
b = np.where(np.array(data_dg > 100))
median = np.median(data_mg[b])
print('Flux median at dispersion > 100 km/s is ' + str(median))
a = np.where(data_dg <= 100)
median1 = np.median(data_mg[a])
print('Flux median at dispersion <= 100 km/s is ' + str(median1))
IndexError: arrays used as indices must be of integer (or boolean) type, line 10

It looks like data_mg and data_dg start with the same shape and you use boolean indexing to keep the values that are not na in each. The trouble is that different values are nan in each array. I would suggest making a combined index that you can use for both arrays.
data_mg = np.ndarray.flatten(data_mg)
data_dg = np.ndarray.flatten(data_dg)
ix_mg = np.logical_not(np.isnan(data_mg))
ix_dg = np.logical_not(np.isnan(data_dg))
ix_combined = np.logical_and(ix_mg, ix_dg)
data_mg = data_mg[ix_combined]
data_dg = data_dg[ix_combined]

First, you could just do the same indexing operation on each array so they'll be of the same shape. I believe that would look something like this:
idx = data_mg / data_err > 2
data_mg = data_mg[idx]
data_df = data_dg[idx)
But the error you're getting may not be due to this. It looks like your error is coming from the line:
data_dg = data_dg[data_mg]
Giving the error:
IndexError: arrays used as indices must be of integer (or boolean) type, line 10
I'm not sure what your intent is here, so I'm not sure what to recommend. If this is you trying to get them to be the same shape, the lines I included above should do that for you.

Python List Append with bumpy array error

I am trying to use list append function to append a list to a list.
But got error shows list indices must be integers or slices, not tuple. Not sure why.
pca_components = range(1,51)
gmm_components = range(1,5)
covariance_types = ['spherical', 'diag', 'tied', 'full']
# Spherical
spherical_results = []
for i in pca_components:
pca_model = PCA(n_components=i)
pca_train = pca_model.fit(train_data).transform(train_data)
for j in gmm_components:
parameters = (i+i)*j*2
if parameters > 50:
pass
else:
gmm_model = GMM(n_components=j, covariance_type='spherical')
gmm_model.fit(pca_train)
pca_test = pca_model.transform(test_data)
predictions = gmm_model.predict(pca_test)
accuracy = np.mean(predictions.ravel() == test_labels.ravel())
accuracy=int(accuracy)
spherical_results.append([accuracy, i,j, parameters])
spher_results = np.array(spherical_results)
max_accuracy = np.amax(spherical_results[:,0])
print(f"highest accuracy score for spherical is {max_accuracy}")

What's the purpose of this line?
spher_results = np.array(spherical_results)
It makes an array from a list. But you don't use spher_results in the following code.

Does NumPy have a function equivalent to Matlab's buffer?

I see there is an array_split and split methods but these are not very handy when you have to split an array of length which is not integer multiple of the chunk size. Moreover, these method’s input is the number of slices rather than the slice size. I need something more like Matlab's buffer method which is more suitable for signal processing.
For example, if I want to buffer a signals to chunks of size 60 I need to do: np.vstack(np.hsplit(x.iloc[0:((len(x)//60)*60)], len(x)//60)) which is cumbersome.

I wrote the following routine to handle the use cases I needed, but I have not implemented/tested for "underlap".
Please feel free to make suggestions for improvement.
def buffer(X, n, p=0, opt=None):
'''Mimic MATLAB routine to generate buffer array
MATLAB docs here: https://se.mathworks.com/help/signal/ref/buffer.html
Parameters
----------
x: ndarray
Signal array
n: int
Number of data segments
p: int
Number of values to overlap
opt: str
Initial condition options. default sets the first `p` values to zero,
while 'nodelay' begins filling the buffer immediately.
Returns
-------
result : (n,n) ndarray
Buffer array created from X
'''
import numpy as np
if opt not in [None, 'nodelay']:
raise ValueError('{} not implemented'.format(opt))
i = 0
first_iter = True
while i < len(X):
if first_iter:
if opt == 'nodelay':
# No zeros at array start
result = X[:n]
i = n
else:
# Start with `p` zeros
result = np.hstack([np.zeros(p), X[:n-p]])
i = n-p
# Make 2D array and pivot
result = np.expand_dims(result, axis=0).T
first_iter = False
continue
# Create next column, add `p` results from last col if given
col = X[i:i+(n-p)]
if p != 0:
col = np.hstack([result[:,-1][-p:], col])
i += n-p
# Append zeros if last row and not length `n`
if len(col) < n:
col = np.hstack([col, np.zeros(n-len(col))])
# Combine result with next row
result = np.hstack([result, np.expand_dims(col, axis=0).T])
return result

def buffer(X = np.array([]), n = 1, p = 0):
#buffers data vector X into length n column vectors with overlap p
#excess data at the end of X is discarded
n = int(n) #length of each data vector
p = int(p) #overlap of data vectors, 0 <= p < n-1
L = len(X) #length of data to be buffered
m = int(np.floor((L-n)/(n-p)) + 1) #number of sample vectors (no padding)
data = np.zeros([n,m]) #initialize data matrix
for startIndex,column in zip(range(0,L-n,n-p),range(0,m)):
data[:,column] = X[startIndex:startIndex + n] #fill in by column
return data

This Keras function may be considered as a Python equivalent of MATLAB Buffer().
See the Sample Code :
import numpy as np
S = np.arange(1,99) #A Demo Array
See Output Here
import tensorflow.keras.preprocessing as kp
list(kp.timeseries_dataset_from_array(S, targets = None,sequence_length=7,sequence_stride=7,batch_size=5))
See the Buffered Array Output Here
Reference : See This

Same as the other answer, but faster.
def buffer(X, n, p=0):
'''
Parameters
----------
x: ndarray
Signal array
n: int
Number of data segments
p: int
Number of values to overlap
Returns
-------
result : (n,m) ndarray
Buffer array created from X
'''
import numpy as np
d = n - p
m = len(X)//d
if m * d != len(X):
m = m + 1
Xn = np.zeros(d*m)
Xn[:len(X)] = X
Xn = np.reshape(Xn,(m,d))
Xne = np.concatenate((Xn,np.zeros((1,d))))
Xn = np.concatenate((Xn,Xne[1:,0:p]), axis = 1)
return np.transpose(Xn[:-1])

ryanjdillon's answer rewritten for significant performance improvement; it appends to a list instead of concatenating arrays, latter which copies the array iteratively and is much slower.
def buffer(x, n, p=0, opt=None):
if opt not in ('nodelay', None):
raise ValueError('{} not implemented'.format(opt))
i = 0
if opt == 'nodelay':
# No zeros at array start
result = x[:n]
i = n
else:
# Start with `p` zeros
result = np.hstack([np.zeros(p), x[:n-p]])
i = n-p
# Make 2D array, cast to list for .append()
result = list(np.expand_dims(result, axis=0))
while i < len(x):
# Create next column, add `p` results from last col if given
col = x[i:i+(n-p)]
if p != 0:
col = np.hstack([result[-1][-p:], col])
# Append zeros if last row and not length `n`
if len(col):
col = np.hstack([col, np.zeros(n - len(col))])
# Combine result with next row
result.append(np.array(col))
i += (n - p)
return np.vstack(result).T

def buffer(X, n, p=0):
'''
Parameters:
x: ndarray, Signal array, input a long vector as raw speech wav
n: int, frame length
p: int, Number of values to overlap
-----------
Returns:
result : (n,m) ndarray, Buffer array created from X
'''
import numpy as np
d = n - p
#print(d)
m = len(X)//d
c = n//d
#print(c)
if m * d != len(X):
m = m + 1
#print(m)
Xn = np.zeros(d*m)
Xn[:len(X)] = X
Xn = np.reshape(Xn,(m,d))
Xn_out = Xn
for i in range(c-1):
Xne = np.concatenate((Xn,np.zeros((i+1,d))))
Xn_out = np.concatenate((Xn_out, Xne[i+1:,:]),axis=1)
#print(Xn_out.shape)
if n-d*c>0:
Xne = np.concatenate((Xn, np.zeros((c,d))))
Xn_out = np.concatenate((Xn_out,Xne[c:,:n-p*c]),axis=1)
return np.transpose(Xn_out)
here is a improved code of Ali Khodabakhsh's sample code which is not work in my cases. Feel free to comment and use it.

Comparing the execution time of the proposed answers, by running
x = np.arange(1,200000)
start = timer()
y = buffer(x,60,20)
end = timer()
print(end-start)
the results are:
Andrzej May, 0.005595300000095449
OverLordGoldDragon, 0.06954789999986133
ryanjdillon, 2.427092700000003

Row, column assignment without for-loop

I wrote a small script to assign values to a numpy array by knowing their row and column coordinates:
gridarray = np.zeros([3,3])
gridarray_counts = np.zeros([3,3])
cols = np.random.random_integers(0,2,15)
rows = np.random.random_integers(0,2,15)
data = np.random.random_integers(0,9,15)
for nn in np.arange(len(data)):
gridarray[rows[nn],cols[nn]] += data[nn]
gridarray_counts[rows[nn],cols[nn]] += 1
In fact, then I know how many values are stored in the same grid cell and what the sum is of them. However, performing this on arrays of lengths 100000+ it is getting quite slow. Is there another way without using a for-loop?
Is an approach similar to this possible? I know this is not working yet.
gridarray[rows,cols] += data
gridarray_counts[rows,cols] += 1

I would use bincount for this, but for now bincount only takes 1darrays so you'll need to write your own ndbincout, something like:
def ndbincount(x, weights=None, shape=None):
if shape is None:
shape = x.max(1) + 1
x = np.ravel_multi_index(x, shape)
out = np.bincount(x, weights, minlength=np.prod(shape))
out.shape = shape
return out
Then you can do:
gridarray = np.zeros([3,3])
cols = np.random.random_integers(0,2,15)
rows = np.random.random_integers(0,2,15)
data = np.random.random_integers(0,9,15)
x = np.vstack([rows, cols])
temp = ndbincount(x, data, gridarray.shape)
gridarray = gridarray + temp
gridarray_counts = ndbincount(x, shape=gridarray.shape)

You can do this directly:
gridarray[(rows,cols)]+=data
gridarray_counts[(rows,cols)]+=1

"shape mismatch" error using numpy in python

I am trying to generate a random array of 0s and 1s, and I am getting the error: shape mismatch: objects cannot be broadcast to a single shape. The error seems to be occurring in the line randints = np.random.binomial(1,p,(n,n)). Here is the function:
import numpy as np
def rand(n,p):
'''Generates a random array subject to parameters p and N.'''
# Create an array using a random binomial with one trial and specified
# parameters.
randints = np.random.binomial(1,p,(n,n))
# Use nested while loops to go through each element of the array
# and assign True to 1 and False to 0.
i = 0
j = 0
rand = np.empty(shape = (n,n),dtype = bool)
while i < n:
while j < n:
if randints[i][j] == 0:
rand[i][j] = False
if randints[i][j] == 1:
rand[i][j] = True
j = j+1
i = i +1
j = 0
# Return the new array.
return rand
print rand
When I run it by itself, it returns <function rand at 0x1d00170>. What does this mean? How should I convert it to an array that can be worked with in other functions?

You needn't go through all of that,
randints = np.random.binomial(1,p,(n,n))
produces your array of 0 and 1 values,
rand_true_false = randints == 1
will produce another array, just with the 1s replaced with True and 0s with False.

Obviously, the answer by #danodonovan is the most Pythonic, but if you really want something more similar to your looping code. Here is an example that fixes the name conflicts and loops more simply.
import numpy as np
def my_rand(n,p):
'''Generates a random array subject to parameters p and N.'''
# Create an array using a random binomial with one trial and specified
# parameters.
randInts = np.random.binomial(1,p,(n,n))
# Use nested while loops to go through each element of the array
# and assign True to 1 and False to 0.
randBool = np.empty(shape = (n,n),dtype = bool)
for i in range(n):
for j in range(n):
if randInts[i][j] == 0:
randBool[i][j] = False
else:
randBool[i][j] = True
return randBool
newRand = my_rand(5,0.3)
print(newRand)