Develop Reference

How can i build matrices in numpy

I want to build a matrix in NumPy in which the items add up to each other. So I have tried to build it with the following code:
StartpointRow = int(input("First number of row?:\n"))
EndpointRow = int(input("Last number of row?:\n"))
StepRow = int(input("Which steps should the row have?:\n"))
StartpointCol = int(input("First number of column?:\n"))
EndpointCol = int(input("Last number of column?:\n"))
StepCol = int(input("Which steps should the column have?:\n"))
x = np.array([[i+j for i in range(StartpointCol, EndpointCol , StepCol)]
for j in range(StartpointRow, EndpointRow , StepRow)])
print(x)
let's say that, for instance, I enter 1,4,1 and 1,4,1. I want the solution to be a matrix like this:
1 2 3 4
2 4 5 6
3 5 6 7
4 6 7 8
Not like that:
2 3 4
3 4 5
4 5 6
or If the user types in: 1,4,1 and 2,4,1.
0 1 2 3 4
2 3 4 5 6
3 4 5 6 7
4 5 6 7 8
not like that:
3 4
4 5
5 6
Could you help me out?

Use np.add.outer:
def build(r_start, r_stop, r_step, c_start, c_stop, c_step):
r = np.arange(r_start, r_stop + 1, r_step)
c = np.arange(c_start, c_stop + 1, c_step)
if r_start == c_start:
ret = np.empty((c.size, r.size), int)
ret[:, 0] = c
ret[0, :] = r
else:
ret = np.empty((c.size + 1, r.size + 1), int)
ret[0, 0] = 0
ret[1:, 0] = c
ret[0, 1:] = r
np.add.outer(ret[1:, 0], ret[0, 1:], out=ret[1:, 1:])
return ret
A little simplification:
def build(r_start, r_stop, r_step, c_start, c_stop, c_step):
r = np.arange(r_start, r_stop + 1, r_step)
c = np.arange(c_start, c_stop + 1, c_step)
ne = int(r_start != c_start)
ret = np.empty((c.size + ne, r.size + ne), int)
ret[0, 0] = 0
ret[ne:, 0] = c
ret[0, ne:] = r
np.add.outer(ret[1:, 0], ret[0, 1:], out=ret[1:, 1:])
return ret
Test:
>>> build(1, 4, 1, 1, 4, 1)
array([[1, 2, 3, 4],
[2, 4, 5, 6],
[3, 5, 6, 7],
[4, 6, 7, 8]])
>>> build(1, 4, 1, 2, 4, 1)
array([[0, 1, 2, 3, 4],
[2, 3, 4, 5, 6],
[3, 4, 5, 6, 7],
[4, 5, 6, 7, 8]])

I think your test cases are wrong.
What I understand you mean is that each row and each column have a starting number, what needs to be done is to add the two and generate the matrix according to step.
If the user types in: 1,4,1 and 1,4,1, what he can get is:
2 3 4 5
3 4 5 6
4 5 6 7
5 6 7 8
And if the user types in: 1,4,1 and 2,4,1, what he can get is:
3 4 5
4 5 6
5 6 7
6 7 8
And my code is:
import numpy as np
StartpointRow = int(input("First number of row?:\n"))
EndpointRow = int(input("Last number of row?:\n"))
StepRow = int(input("Which steps should the row have?:\n"))
StartpointCol = int(input("First number of column?:\n"))
EndpointCol = int(input("Last number of column?:\n"))
StepCol = int(input("Which steps should the column have?:\n"))
x = np.array([[i+j for i in range(StartpointCol, EndpointCol + 1 , StepCol)]
for j in range(StartpointRow, EndpointRow + 1, StepRow)])
print(x)

Comparing and altering values depending on indexes Numpy Python

I have 3 arrays down below a and b combine to make a_and_b. a is multiplied by a_multiplier and b gets multiplied by b_multiplier. I am trying to make a ternary statement where it checks the indexes of a_and_b matches with either a or b then it does the necessary multiplications with a_multiplier or b_multiplier. How would I be able to modify a_and_b ternary function so that the Expected Output works?
import numpy as np
a_multiplier = 3
b_multiplier = 5
a = np.array([5,32,1,4])
b = np.array([1,5,11,3])
a_and_b = np.array([5,1,32,5,1,11,4,3])
a_and_b = a_and_b[::2] * 3,a_and_b[1::2] * 5 if a_and_b[::2] == a
else a_and_b[::2] * 5,a_and_b[1::2] * 3
Expected Output:
[15, 5, 96, 25, 3, 55, 12, 15]

IIUC, you could do directly:
import numpy as np
a_multiplier = 3
b_multiplier = 5
a = np.array([5, 32, 1, 4])
b = np.array([1, 5, 11, 3])
res = np.vstack([a * a_multiplier, b * b_multiplier]).flatten("F")
print(res)
Output
[15 5 96 25 3 55 12 15]

Iterative summation

I'm trying to write a python code that allows me to iteratively sum up the average values of three elements of a list, starting with the third element and its two predecessors. Let me give you an example:
list = [1, 2, 3, 4, 5, 6, 7]
I want to calculate the following:
sum_of_average_values = sum(1, 2, 3)/3 + sum(2, 3, 4)/3 + sum(3, 4, 5)/3 + sum(4, 5, 6)/3 + sum(5, 6, 7)/3
Since I'm quite new to programming I couldn't find an effective way of putting this into a function.

You can do in this way:
a = [1,2,3,4,5,6,7]
sum_of_average_values = 0
for i in range(0,len(a)-2):
sum_of_average_values += sum(a[i:i+2])/3
print(sum_of_average_values)

You could use rolling from pandas.
import pandas as pd
num_list = [1, 2, 3, 4, 5, 6, 7]
average_sum = sum(pd.Series(num_list).rolling(3).mean().dropna())
print(average_sum)

Many ways to achieve this, one way is recursively.
The function averages the last three elements of a list and adds the result to the result generated by the function with a list lacking the last element. Continues like this until the list is shorter than 3.
def fn(l):
if len(l) < 3:
return 0
return sum(l[-3:])/3 + fn(l[:-1])
print(fn([1, 2, 3, 4, 5, 6, 7]))

Another solution where you can specify the amount of elements you want to sum up and average:
l = [1, 2, 3, 4, 5, 6, 7]
def sum_avg(l, n):
res = 0
for i in range(n-1, len(l)):
res += sum([l[j] for j in range(i, i-n, -1)])/n
return res
print(sum_avg(l, 3))
--> 20.0

Mathematically, this would could be obtain by averaging the sums of 3 sublists:
L = [1, 2, 3, 4, 5, 6, 7]
r = (sum(L) + sum(L[1:-1]) + sum(L[2:-2]))/3 # 20.0
and can be generalized to a window size of w:
w = 3
r = sum(sum(L[p:-p or None]) for p in range(w)) / w
It can also be implemented without the overhead of generating sublists by using item positions to determine the number of times they are added to the total:
r = sum(n*min(i+1,len(L)-i,w) for i,n in enumerate(L)) / w
This would be the most memory-efficient of the 3 methods because it use an iterator to feed data to the sum function and only goes through the data once.
Detailed explanation:
Since all the averages that are added together are a division by 3, we can produce the total sum and divide by 3 at the end
the number at the first and last positions are added once
the number at the second and penultimate positions are added twice
The numbers from the third position up to the antepenultimate will be added 3 times
visually:
(1 + 2 + 3) / 3
(2 + 3 + 4) / 3
(3 + 4 + 5) / 3
(4 + 5 + 6) / 3
(5 + 6 + 7) / 3
(1x1 + 2x2 + 3x3 + 4x3 + 5x3 + 6x2 + 7x1) / 3 = 20.0
n = 1 2 3 4 5 6 7 # value
* = 1 2 3 3 3 2 1 # multiplier (times added)
-------------------------
(2, 4, 9, 12, 15, 12, 7) / 3 = 20.0
i = 0 1 2 3 4 5 6 # index
1 2 3 3 3 2 1 # min(i+1,len(L)-i,w) = multiplier

You can do in one line using list comprehension as:
n = 3
avg = sum( [ sum(lst[i:i+n])/n for i in range(0, len(lst) - (n - 1)) ] )
print(avg) # 20.0

Counting the number of consecutive values that meets a condition (Pandas Dataframe)

So I created this post regarding my problem 2 days ago and got an answer thankfully.
I have a data made of 20 rows and 2500 columns. Each column is a unique product and rows are time series, results of measurements. Therefore each product is measured 20 times and there are 2500 products.
This time I want to know for how many consecutive rows my measurement result can stay above a specific threshold.
AKA: I want to count the number of consecutive values that is above a value, let's say 5.
A = [1, 2, 6, 8, 7, 3, 2, 3, 6, 10, 2, 1, 0, 2]
We have these values in bold and according to what I defined above, I should get NumofConsFeature = 3 as the result. (Getting the max if there are more than 1 series that meets the condition)
I thought of filtering using .gt, then getting the indexes and using a loop afterwards in order to detect the consecutive index numbers but couldn't make it work.
In 2nd phase, I'd like to know the index of the first value of my consecutive series. For the above example, that would be 3.
But I have no idea of how for this one.
Thanks in advance.

Here's another answer using only Pandas functions:
A = [1, 2, 6, 8, 7, 3, 2, 3, 6, 10, 2, 1, 0, 2]
a = pd.DataFrame(A, columns = ['foo'])
a['is_large'] = (a.foo > 5)
a['crossing'] = (a.is_large != a.is_large.shift()).cumsum()
a['count'] = a.groupby(['is_large', 'crossing']).cumcount(ascending=False) + 1
a.loc[a.is_large == False, 'count'] = 0
which gives
foo is_large crossing count
0 1 False 1 0
1 2 False 1 0
2 6 True 2 3
3 8 True 2 2
4 7 True 2 1
5 3 False 3 0
6 2 False 3 0
7 3 False 3 0
8 6 True 4 2
9 10 True 4 1
10 2 False 5 0
11 1 False 5 0
12 0 False 5 0
13 2 False 5 0
From there on you can easily find the maximum and its index.

There is simple way to do that.
Lets say your list is like: A = [1, 2, 6, 8, 7, 6, 8, 3, 2, 3, 6, 10,6,7,8, 2, 1, 0, 2]
And you want to find how many consecutive series that has values bigger than 6 and length of 5. For instance, here your answer is 2. There is two series that has values bigger than 6 and length of the series are 5. In python and pandas we do that like below:
condition = (df.wanted_row > 6) & \
(df.wanted_row.shift(-1) > 6) & \
(df.wanted_row.shift(-2) > 6) & \
(df.wanted_row.shift(-3) > 6) & \
(df.wanted_row.shift(-4) > 6)
consecutive_count = df[condition].count().head(1)[0]

Here's one with maxisland_start_len_mask -
# https://stackoverflow.com/a/52718782/ #Divakar
def maxisland_start_len_mask(a, fillna_index = -1, fillna_len = 0):
# a is a boolean array
pad = np.zeros(a.shape[1],dtype=bool)
mask = np.vstack((pad, a, pad))
mask_step = mask[1:] != mask[:-1]
idx = np.flatnonzero(mask_step.T)
island_starts = idx[::2]
island_lens = idx[1::2] - idx[::2]
n_islands_percol = mask_step.sum(0)//2
bins = np.repeat(np.arange(a.shape[1]),n_islands_percol)
scale = island_lens.max()+1
scaled_idx = np.argsort(scale*bins + island_lens)
grp_shift_idx = np.r_[0,n_islands_percol.cumsum()]
max_island_starts = island_starts[scaled_idx[grp_shift_idx[1:]-1]]
max_island_percol_start = max_island_starts%(a.shape[0]+1)
valid = n_islands_percol!=0
cut_idx = grp_shift_idx[:-1][valid]
max_island_percol_len = np.maximum.reduceat(island_lens, cut_idx)
out_len = np.full(a.shape[1], fillna_len, dtype=int)
out_len[valid] = max_island_percol_len
out_index = np.where(valid,max_island_percol_start,fillna_index)
return out_index, out_len
def maxisland_start_len(a, trigger_val, comp_func=np.greater):
# a is 2D array as the data
mask = comp_func(a,trigger_val)
return maxisland_start_len_mask(mask, fillna_index = -1, fillna_len = 0)
Sample run -
In [169]: a
Out[169]:
array([[ 1, 0, 3],
[ 2, 7, 3],
[ 6, 8, 4],
[ 8, 6, 8],
[ 7, 1, 6],
[ 3, 7, 8],
[ 2, 5, 8],
[ 3, 3, 0],
[ 6, 5, 0],
[10, 3, 8],
[ 2, 3, 3],
[ 1, 7, 0],
[ 0, 0, 4],
[ 2, 3, 2]])
# Per column results
In [170]: row_index, length = maxisland_start_len(a, 5)
In [172]: row_index
Out[172]: array([2, 1, 3])
In [173]: length
Out[173]: array([3, 3, 4])

You can apply diff() on your Series, and then just count the number of consecutive entries where the difference is 1 and the actual value is above your cutoff. The largest count is the maximum number of consecutive values.
First compute diff():
df = pd.DataFrame({"a":[1, 2, 6, 7, 8, 3, 2, 3, 6, 10, 2, 1, 0, 2]})
df['b'] = df.a.diff()
df
a b
0 1 NaN
1 2 1.0
2 6 4.0
3 7 1.0
4 8 1.0
5 3 -5.0
6 2 -1.0
7 3 1.0
8 6 3.0
9 10 4.0
10 2 -8.0
11 1 -1.0
12 0 -1.0
13 2 2.0
Now count consecutive sequences:
above = 5
n_consec = 1
max_n_consec = 1
for a, b in df.values[1:]:
if (a > above) & (b == 1):
n_consec += 1
else: # check for new max, then start again from 1
max_n_consec = max(n_consec, max_n_consec)
n_consec = 1
max_n_consec
3

Here's how I did it using numpy:
import pandas as pd
import numpy as np
df = pd.DataFrame({"a":[1, 2, 6, 7, 8, 3, 2, 3, 6, 10, 2, 1, 0, 2]})
consecutive_steps = 2
marginal_price = 5
assertions = [(df.loc[:, "a"].shift(-i) < marginal_price) for i in range(consecutive_steps)]
condition = np.all(assertions, axis=0)
consecutive_count = df.loc[condition, :].count()
print(consecutive_count)
which yields 6.

Tensorflow efficient overlap-add

I'm trying to implement overlap-add in Tensorflow but I'm struggling to convert the numpy output_seq[start:end] += chunk to Tensorflow. Right now I'm output_seq = output_seq + tf.pad(chunk, [[start, length - end]]) but that's really slow on long sequences.
I also have a hunch there might be tricks you can do with gather/scatter, but I can't quite figure it out. Below is my brute force attempt:
import tensorflow as tf
input = [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
def overlap_add(overlap):
with tf.Graph().as_default(), tf.Session() as sess:
x = tf.constant(input)
num_chunks = tf.shape(x)[0]
chunk_size = tf.shape(x)[1]
hop_length = chunk_size - overlap
out_len = chunk_size + hop_length * (num_chunks - 1)
y = tf.zeros((out_len,), dtype=tf.int32)
def body(i, y):
j = i * hop_length
padding = [[j, out_len - (j + chunk_size)]]
chunk = x[i]
y = y + tf.pad(chunk, padding)
return (i + 1, y)
i = tf.constant(0)
i, y = tf.while_loop(
cond=lambda i, _: tf.less(i, num_chunks),
body=body,
loop_vars=[i, y])
return sess.run(y)
for i in range(4):
print 'overlap_add(%d): %s' % (i, overlap_add(i))
# overlap_add(0): [ 1 2 3 4 5 6 7 8 9 10 11 12]
# overlap_add(1): [ 1 2 3 9 6 7 17 10 11 12]
# overlap_add(2): [ 1 2 8 10 16 18 11 12]
# overlap_add(3): [ 1 7 18 21 19 12]

UPDATE: There's now an overlap_and_add function in Tensorflow itself.
OLD ANSWER:
Trawled through the docs and found unsorted_segment_sum:
import tensorflow as tf
input = [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
def tf_repeat(a, repeats):
return tf.reshape(tf.tile(tf.reshape(a, [-1, 1]),
[1, repeats]), [-1])
def overlap_add(overlap):
with tf.Graph().as_default(), tf.Session() as sess:
x = tf.constant(input)
x_flat = tf.reshape(x, [-1])
num_chunks = tf.shape(x)[0]
chunk_size = tf.shape(x)[1]
hop_len = chunk_size - overlap
flat_len = num_chunks * chunk_size
out_len = chunk_size + hop_len * (num_chunks - 1)
# e.g. [0,1,2,3, 2,3,4,5, 4,5,6,7] for overlap == 2
indexes = tf.range(flat_len) - tf_repeat(tf.range(num_chunks), chunk_size) * overlap
return sess.run(tf.unsorted_segment_sum(x_flat, indexes, out_len))
for i in range(4):
print 'overlap_add(%d): %s' % (i, overlap_add(i))
# overlap_add(0): [ 1 2 3 4 5 6 7 8 9 10 11 12]
# overlap_add(1): [ 1 2 3 9 6 7 17 10 11 12]
# overlap_add(2): [ 1 2 8 10 16 18 11 12]
# overlap_add(3): [ 1 7 18 21 19 12]

You can use slices in Tensorflow as well:
a[1:3].assign(a[1:3] + b[1:3]).eval()
For some reason assign_add is not implemented. It seems like a bug to me.
a[1:3].assign_add(b[1:3]).eval() # Doesn't work