I have two 2d arrays, one containing float values, one containing bool. I want to create an array containing the mean values of the first matrix for each column considering only the values corresponding to False in the second matrix.
For example:
A = [[1 3 5]
[2 4 6]
[3 1 0]]
B = [[True False False]
[False False False]
[True True False]]
result = [2, 3.5, 3.67]
Where B is False, keep the value of A, make it NaN otherwise and then use the nanmean function which ignores NaN's for operations.
np.nanmean(np.where(~B, A, np.nan), axis=0)
>>> array([2. , 3.5 , 3.66666667])
Using numpy.mean using where argument to specify elements to include in the mean.
np.mean(A, where = ~B, axis = 0)
>>> [2. 3.5 3.66666667]
A = [[1, 3, 5],
[2, 4, 6],
[3, 1, 0]]
B = [[True, False, False],
[False, False, False],
[True, True, False]]
sums = [0]*len(A[0])
amounts = [0]*len(A[0])
for i in range(0, len(A)):
for j in range(0, len(A[0])):
sums[j] = sums[j] + (A[i][j] if not B[i][j] else 0)
amounts[j] = amounts[j] + (1 if not B[i][j] else 0)
result = [sums[i]/amounts[i] for i in range(0, len(sums))]
print(result)
There may be some fancy numpy trick for this, but I think using a list comprehension to construct a new array is the most straightforward.
result = np.array([a_col[~b_col].mean() for a_col, b_col in zip(A.T,B.T)])
To follow better, this is what the line does expanded out:
result=[]
for i in range(len(A)):
new_col = A[:,i][~B[:,i]]
result.append(new_col.mean())
You could also use a masked array:
import numpy as np
result = np.ma.array(A, mask=B).mean(axis=0).filled(fill_value=0)
# Output:
# array([2. , 3.5 , 3.66666667])
which has the advantage of being able to supply a fill_value for when every element in some column in B is True.
Assuming I have n = 3 lists of same length for example:
R1 = [7,5,8,6,0,6,7]
R2 = [8,0,2,2,0,2,2]
R3 = [1,7,5,9,0,9,9]
I need to find the first index t that verifies the n = 3 following conditions for a period p = 2.
Edit: the meaning of period p is the number of consecutive "boxes".
R1[t] >= 5, R1[t+1] >= 5. Here t +p -1 = t+1, we need to only verify for two boxes t and t+1. If p was equal to 3 we will need to verify for t, t+1 and t+2. Note that It's always the same number for which we test, we always test if it's greater than 5 for every index. The condition is always the same for all the "boxes".
R2[t] >= 2, R2[t+1] >= 2
R3[t] >= 9, R3[t+1] >= 9
In total there is 3 * p conditions.
Here the t I am looking for is 5 (indexing is starting from 0).
The basic way to do this is by looping on all the indexes using a for loop. If the condition is found for some index t we store it in some local variable temp and we verify the conditions still hold for every element whose index is between t+1 and t+p -1. If while checking, we find an index that does not satisfy a condition, we forget about the temp and we keep going.
What is the most efficient way to do this in Python if I have large lists (like of 10000 elements)? Is there a more efficient way than the for loop?
Since all your conditions are the same (>=), we could leverage this.
This solution will work for any number of conditions and any size of analysis window, and no for loop is used.
You have an array:
>>> R = np.array([R1, R2, R3]).T
>>> R
array([[7, 8, 1],
[5, 0, 7],
[8, 2, 5],
[6, 2, 9],
[0, 0, 0],
[6, 2, 9],
[7, 2, 9]]
and you have thresholds:
>>> thresholds = [5, 2, 9]
So you can check where the conditions are met:
>>> R >= thresholds
array([[ True, True, False],
[ True, False, False],
[ True, True, False],
[ True, True, True],
[False, False, False],
[ True, True, True],
[ True, True, True]])
And where they all met at the same time:
>>> R_cond = np.all(R >= thresholds, axis=1)
>>> R_cond
array([False, False, False, True, False, True, True])
From there, you want the conditions to be met for a given window.
We'll use the fact that booleans can sum together, and convolution to apply the window:
>>> win_size = 2
>>> R_conv = np.convolve(R_cond, np.ones(win_size), mode="valid")
>>> R_conv
array([0., 0., 1., 1., 1., 2.])
The resulting array will have values equal to win_size at the indices where all conditions are met on the window range.
So let's retrieve the first of those indices:
>>> index = np.where(R_conv == win_size)[0][0]
>>> index
5
If such an index doesn't exist, it will raise an IndexError, I'm letting you handle that.
So, as a one-liner function, it gives:
def idx_conditions(arr, thresholds, win_size, condition):
return np.where(
np.convolve(
np.all(condition(arr, thresholds), axis=1),
np.ones(win_size),
mode="valid"
)
== win_size
)[0][0]
I added the condition as an argument to the function, to be more general.
>>> from operator import ge
>>> idx_conditions(R, thresholds, win_size, ge)
5
This could be a way:
R1 = [7,5,8,6,0,6,7]
R2 = [8,0,2,2,0,2,2]
R3 = [1,7,5,9,0,9,9]
for i,inext in zip(range(len(R1)),range(len(R1))[1:]):
if (R1[i]>=5 and R1[inext]>=5)&(R2[i]>=2 and R2[inext]>=2)&(R3[i]>=9 and R3[inext]>=9):
print(i)
Output:
5
Edit: Generalization could be:
def foo(ls,conditions):
index=0
for i,inext in zip(range(len(R1)),range(len(R1))[1:]):
if all((ls[j][i]>=conditions[j] and ls[j][inext]>=conditions[j]) for j in range(len(ls))):
index=i
return index
R1 = [7,5,8,6,0,6,7]
R2 = [8,0,2,2,0,2,2]
R3 = [1,7,5,9,0,9,9]
R4 = [1,7,5,9,0,1,1]
R5 = [1,7,5,9,0,3,3]
conditions=[5,2,9,1,3]
ls=[R1,R2,R3,R4,R5]
print(foo(ls,conditions))
Output:
5
And, maybe if the arrays match the conditions multiple times, you could return a list of the indexes:
def foo(ls,conditions):
index=[]
for i,inext in zip(range(len(R1)),range(len(R1))[1:]):
if all((ls[j][i]>=conditions[j] and ls[j][inext]>=conditions[j]) for j in range(len(ls))):
print(i)
index.append(i)
return index
R1 = [6,7,8,6,0,6,7]
R2 = [2,2,2,2,0,2,2]
R3 = [9,9,5,9,0,9,9]
R4 = [1,1,5,9,0,1,1]
R5 = [3,3,5,9,0,3,3]
conditions=[5,2,9,1,3]
ls=[R1,R2,R3,R4,R5]
print(foo(ls,conditions))
Output:
[0,5]
Here is a solution using numpy ,without for loops:
import numpy as np
R1 = np.array([7,5,8,6,0,6,7])
R2 = np.array([8,0,2,2,0,2,2])
R3 = np.array([1,7,5,9,0,9,9])
a = np.logical_and(np.logical_and(R1>=5,R2>=2),R3>=9)
np.where(np.logical_and(a[:-1],a[1:]))[0].item()
ouput
5
Edit:
Generalization
Say you have a list of lists R and a list of conditions c:
R = [[7,5,8,6,0,6,7],
[8,0,2,2,0,2,2],
[1,7,5,9,0,9,9]]
c = [5,2,9]
First we convert them to numpy arrays. the reshape(-1,1) converts c to a column matrix so that we can use pythons broadcasting feature in the >= operator
R = np.array(R)
c = np.array(c).reshape(-1,1)
R>=c
output:
array([[ True, True, True, True, False, True, True],
[ True, False, True, True, False, True, True],
[False, False, False, True, False, True, True]])
then we perform logical & operation between all rows using reduce function
a = np.logical_and.reduce(R>=c)
a
output:
array([False, False, False, True, False, True, True])
next we create two arrays by removing first and last element of a and perform a logical & between them which shows which two subsequent elements satisfied the conditions in all lists:
np.logical_and(a[:-1],a[1:])
output:
array([False, False, False, False, False, True])
now np.where just shows the index of the True element
np.where(np.logical_and(a[:-1],a[1:]))[0].item()
output:
5
I have this 3x3 matrix:
a=array([[ 1, 11, 5],
[ 3, 9, 9],
[ 5, 7, -3]])
I need to mask the minimum values in each row in order to calculate the mean of each row discarding the minimum values. Is there a general solution?
I have tried with
a_masked=np.ma.masked_where(a==np.ma.min(a,axis=1),a)
Which masks the minimum value in first and third row, but not the second row?
I would appreciate any help. Thanks!
The issue is because the comparison a == a.min(axis=1) is comparing each column to the minimum value of each row rather than comparing each row to the minimum values. This is because a.min(axis=1) returns a vector rather than a matrix which behaves similarly to an Nx1 array. As such, when broadcasting, the == operator performs the operation in a column-wise fashion to match dimensions.
a == a.min(axis=1)
# array([[ True, False, False],
# [False, False, False],
# [False, False, True]], dtype=bool)
One potential way to fix this is to resize the result of a.min(axis=1) into column vector (e.g. a 3 x 1 2D array).
a == np.resize(a.min(axis=1), [a.shape[0],1])
# array([[ True, False, False],
# [ True, False, False],
# [False, False, True]], dtype=bool)
Or more simply as #ColonelBeuvel has shown:
a == a.min(axis=1)[:,None]
Now applying this to your entire line of code.
a_masked = np.ma.masked_where(a == np.resize(a.min(axis=1),[a.shape[0],1]), a)
# masked_array(data =
# [[-- 11 5]
# [-- 9 9]
# [5 7 --]],
# mask =
# [[ True False False]
# [ True False False]
# [False False True]],
# fill_value = 999999)
What is with the min() function?
For every Row just do min(row) and it gives you the minimum of this list in your Case a row. Simply append this minimum in a list for all Minimum.
minList=[]
for i in array:
minList.append(min(i))
I have a numpy array arr with negative double elements. It is shaped (1000,1000). As the elements are complex, we use arr.real to only evaluate the real part.
I first tried
for i in arr.real:
if i < 0:
print(i)
This gave the following ValueError:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
However, if I try
for i in arr.real:
if i.any() < 0:
print(i)
or
for i in arr.real:
if i.all() < 0:
print(i)
there is no output. Nothing is printed, even though negative values do exist.
How do I rectify this? What am I misunderstanding?
EDIT:
for i in arr.real:
print(i[i<0])
does work. However, how does one search for two conditions? For example,
i < 0 and i > -1e-12
Since arr is 2d, iteration gives you the rows, not the elements.
Make a sample array:
In [347]: arr=np.arange(16).reshape(4,4)-10
In [348]: arr
Out[348]:
array([[-10, -9, -8, -7],
[ -6, -5, -4, -3],
[ -2, -1, 0, 1],
[ 2, 3, 4, 5]])
Iterate with some prints:
In [350]: for i in arr:
print(i)
print(i<0)
print((i<0).any())
.....:
result:
[-10 -9 -8 -7]
[ True True True True]
True
[-6 -5 -4 -3]
[ True True True True]
True
[-2 -1 0 1]
[ True True False False]
True
[2 3 4 5]
[False False False False]
False
The ValueError results when you try to use that boolean array np.array([True, True, False, False]) in an if statement. Applying all or any to the array reduces it to one scalar True/False value, which works in the if statement.
You can apply the negative test to the whole array, and apply all/any to rows (or columns) - without iteration:
In [351]: arr<0
Out[351]:
array([[ True, True, True, True],
[ True, True, True, True],
[ True, True, False, False],
[False, False, False, False]], dtype=bool)
In [352]: (arr<0).any(axis=1)
Out[352]: array([ True, True, True, False], dtype=bool)
In [353]: (arr<0).all(axis=1)
Out[353]: array([ True, True, False, False], dtype=bool)
to get the non-negative values in the array, you can use this boolean mask (or its negative):
In [354]: arr[arr>=0]
Out[354]: array([0, 1, 2, 3, 4, 5])
Because there are different numbers of valid values in each row it can't give you a 2d array.
But you can go back to iteration to get a list of values for each row. Here I use a list comprehension to do the iteration.
In [355]: [a[a>=0] for a in arr]
Out[355]:
[array([], dtype=int32),
array([], dtype=int32),
array([0, 1]),
array([2, 3, 4, 5])]
Have you tried lambda expressions?
for k in arr:
print filter(lambda x: x < 0, k)
if your "array" is a dictionary with tuple keys you can use lambda alike this:
d = {(0,0):-3,(0,1):3,(0,2):-3.7,(0,3):0,
(1,0):30,(1,1):-12,(1,2):-0.1,(1,3):2.5,}
keys = filter(lambda x: d[x] < 0, [k for k in d])
print [d[keys[n]] for n in range(0,len(keys))]
perhaps to actually answer your question this will help,
import itertools as it
aK = it.product(range(0,len(arr)),range(0,len(arr)))
negKeys = filter(lambda x: arr[x[0]][x[1]] < 0, [k for k in ak])
negVals = [arr[k[0]][k[1]] for k in negKeys]
It seems one way to do this is:
for row in arr.real:
for i in row:
if (i<0 and i<-1e-10):
print(i)
With a dataframe like one shown below:
A B
0 True False
1 True False
2 True False
3 False True
4 False False
5 True False
I need to get a list of the indexes for alternating booleans between column a and b. So the first time column A shows True, I can add 0 to the list. Now I switch to column B and find the next index after 0 that shows True which is 3. Then I switch back to column A to add the next index to show True after index 3. In the end I will have a list with the values [0, 3, 5]. I'm currently building these lists by iterating through the rows in a for loop and with if statements. I don't think this is the most efficient way this can be done. Any help on the "correct" way to do this will be much appreciated. Thanks!
One way could be something like this:
In [3]:
df['C'] = df.A.astype(int) - df.B.astype(int)
df['D'] = df[['C']].apply(lambda x: (x != x.shift()).astype(int).cumsum())
df[(df.C == 1) | (df.C == -1)].groupby('D').head(1).index
Out[3]:
Int64Index([0, 3, 5], dtype='int64')
I haven't tested it extensively, but it works with the samples you provided, including this one:
df = pd.DataFrame({'A': [True, True, False, True, False, False, True],
'B': [False, False, True, False, False, True, False]})
In [4]:
df['C'] = df.A.astype(int) - df.B.astype(int)
df['D'] = df[['C']].apply(lambda x: (x != x.shift()).astype(int).cumsum())
df[(df.C == 1) | (df.C == -1)].groupby('D').head(1).index.tolist()
Out[4]:
[0, 2, 3, 5, 6]
Is this better?
inputList=[(True,False),(True,False),(True,False),
(False,True),(False,False),(True,False)]
l=list()
toggle=0
for i in range(0,len(inputList)):
if (inputList[i][toggle%2]==True):
l.append(i)
toggle+=1
print l