Develop Reference

How to Find the Max Plot of Land From a 2D Array?

Sorry if the title was a little confusing, I didn't know what to call it. However, I'm still new to programming and I'm stuck on this coding problem that I just have no idea where to start.
Here is the summarized version of the problem:
I have a randomized plot of land, lets just call the variables x and y. This plot of land is a 2D array of all numbers that can be negative or positive. Now, there will be another, smaller plot of randomized numbers, lets call them width, height. With these new variables I need to find the greatest number from the x,y array that is width, height in size.
All numbers will be valid integers.
x ≥ width > 0
y ≥ height > 0
I will need to output the largest sum of land in the x y plot that is width, height in size.
Here is an example
3 - randomly picked y value
4 - randomly picked x value
2 - randomly picked height
1 - randomly picked width
1 2 3 4
-1 0 -1 9
-4 1 -2 7
Now, you can see from the example that the output will be 16, because the biggest 1x2 plot in the 4x3 plot is 16. I was wondering if anyone could point me in the right direction and give me tips on where to start. I have tried researching this, but it has led nowhere because I have no idea what to look up.

A summed-area table seems to be an interesting way to tackle this problem. If I'm not mistaken such an algorithm would be linear in the number of cells (x*y).
The basic idea of a summed-area table is that the sum of a subparcel can be calculated by adding the values for two corners and subtracting the values of the opposite corners, as explained in the Wikipedia article.
Numpy's cumsum helps to quickly create the summed-area table. Maybe there is also a numpy way to calculate the areas?
Here's my sample code (note that numpy first indexes the vertical direction, and then the horizontal). The tests inside the loop could be skipped if we added an extra row and extra column of zeros (but would make the code slightly more difficult to understand).
import numpy as np
def find_highest_area_sum(parcel, x, y, width, height):
sums = np.cumsum(np.cumsum(parcel, axis=0), axis=1)
areas = np.zeros((y - height + 1, x - width + 1), dtype=sums.dtype)
print("Given parcel:")
print(parcel)
print("Cumulative area sums:")
print(sums)
for i in range(x - width + 1):
for j in range(y - height + 1):
areas[j, i] = sums[j + height - 1, i + width - 1]
if i > 0:
areas[j, i] -= sums[j + height - 1, i - 1]
if j > 0:
areas[j, i] -= sums[j - 1, i + width - 1]
if i > 0 and j > 0:
areas[j, i] += sums[j - 1, i - 1]
print("Areas of each subparcel:")
print(areas)
ind_highest = np.unravel_index(np.argmax(areas), areas.shape)
print(f'The highest area sum is {areas[ind_highest]} at pos ({ind_highest[1]}, {ind_highest[0]}) to pos ({ind_highest[1] + width - 1}, {ind_highest[0] + height - 1}) ')
x, y = 4, 3
width, height = 1, 2
parcel = np.array([[1, 2, 3, 4],
[-1, 0, -1, 9],
[-4, 1, -2, 7]])
find_highest_area_sum(parcel, x, y, width=1, height=2)
x = 12
y = 20
parcel = np.random.randint(-10, 20, (y, x))
find_highest_area_sum(parcel, x, y, width=10, height=12)
Output of the first part:
Given parcel:
[[ 1 2 3 4]
[-1 0 -1 9]
[-4 1 -2 7]]
Cumulative area sums:
[[ 1 3 6 10]
[ 0 2 4 17]
[-4 -1 -1 19]]
Areas of each subparcel:
[[ 0 2 2 13]
[-5 1 -3 16]]
The highest area sum is 16 at pos (3, 1) to pos (3, 2)

Get top values and positions of items in list of lists

I have used sklearn to fit and predict a model, but I want to have the top 5 predictions (in terms of probabilities) per item.
So I used predict_proba, which gave me a list of lists like:
probabilities = [[0.8,0.15,0.5,0,0],[0.4,0.6,0,0,0],[0,0,0,0,1]]
What I want to do, is loop over this list of lists to give me an overview of each prediction made, along with its position in the list (which represents the classes).
When using [i for i, j in enumerate(predicted_proba[0]) if j > 0] it returns me [0],[1] , which is what I want for the complete list of lists (and if possible also with the probability next to it).
When trying to use a for-loop over the above code, it returns an IndexError.

Something like this:
probabilities = [[0.8, 0.15, 0.5, 0, 0], [0.4, 0.6, 0, 0, 0], [0, 0, 0, 0, 1]]
for list in range(0,len(probabilities)):
print("Iteration_number:", list)
for index, prob in enumerate(probabilities[list]):
print("index", index, "=", prob)
Results in:
Iteration_number: 0
index 0 = 0.8
index 1 = 0.15
index 2 = 0.5
index 3 = 0
index 4 = 0
Iteration_number: 1
index 0 = 0.4
index 1 = 0.6
index 2 = 0
index 3 = 0
index 4 = 0
Iteration_number: 2
index 0 = 0
index 1 = 0
index 2 = 0
index 3 = 0
index 4 = 1

for i in predicted_proba:
for index, value in enumerate(i):
if value > 0:
print(index)
Hope this helps.

How to sort rows of a binary-valued array as if they were long binary numbers?

There is a 2D numpy array of about 500000 rows by 512 values each row:
[
[1,0,1,...,0,0,1], # 512 1's or 0's
[0,1,0,...,0,1,1],
...
[0,0,1,...,1,0,1], # row number 500000
]
How to sort the rows ascending as if each row is a long 512-bit integer?
[
[0,0,1,...,1,0,1],
[0,1,0,...,0,1,1],
[1,0,1,...,0,0,1],
...
]

Instead of converting to strings you can also use a void view (as from #Jaime here) of the data and argsort by that.
def sort_bin(b):
b_view = np.ascontiguousarray(b).view(np.dtype((np.void, b.dtype.itemsize * b.shape[1])))
return b[np.argsort(b_view.ravel())] #as per Divakar's suggestion
Testing
np.random.seed(0)
b = np.random.randint(0, 2, (10,5))
print(b)
print(sort_bin(b))
[[0 1 1 0 1]
[1 1 1 1 1]
[1 0 0 1 0]
...,
[1 0 1 1 0]
[0 1 0 1 1]
[1 1 1 0 1]]
[[0 0 0 0 1]
[0 1 0 1 1]
[0 1 1 0 0]
...,
[1 1 1 0 1]
[1 1 1 1 0]
[1 1 1 1 1]]
Should be much faster and less memory-intensive since b_view is just a view into b
t = np.random.randint(0,2,(2000,512))
%timeit sort_bin(t)
100 loops, best of 3: 3.09 ms per loop
%timeit np.array([[int(i) for i in r] for r in np.sort(np.apply_along_axis(lambda r: ''.join([str(c) for c in r]), 0, t))])
1 loop, best of 3: 3.29 s per loop
About 1000x faster actually

You could sort them in a stable way 512 times, starting with the right-most bit first.
Sort by last bit
Sort by second-last bit, stable (to not mess up results of previous sort)
...
...
Sort by first bit, stable
A smaller example: assume you want to sort these three 2-bit numbers by bits:
11
01
00
In the first step, you sort by the right bit, resulting in:
00
11
01
Now you sort by the first bit, in this case we have two 0s in that column. If your sorting algorithm is not stable it would be allowed to put these equal items in any order in the result, that could cause 01 to appear before 00 which we do not want, so we use a stable sort, keeping the relative order of equal items, for the first column, resulting in the desired:
00
01
11

Creating a string of each row and then applying np.sort()
So if we have an array to test on:
a = np.array([[1,0,0,0],[0,0,0,0],[1,1,1,1],[0,0,1,1]])
We can create strings of each row by using np.apply_along_axis:
a = np.apply_along_axis(lambda r: ''.join([str(c) for c in r]), 0, a)
which would make a now:
array(['1010', '0010', '0011', '0011'], dtype='<U4')
and so now we can sort the strings with np.sort():
a = np.sort(a)
making a:
array(['0010', '0011', '0011', '1010'], dtype='<U4')
we can then convert back to the original format with:
a = np.array([[int(i) for i in r] for r in a])
which makes a:
array([[0, 0, 1, 0],
[0, 0, 1, 1],
[0, 0, 1, 1],
[1, 0, 1, 0]])
And if you wanted to cram this all into one line:
a = np.array([[int(i) for i in r] for r in np.sort(np.apply_along_axis(lambda r: ''.join([str(c) for c in r]), 0, a))])

This is slow but does the job.
def sort_col(arr, col_num=0):
# if we have sorted over all columns return array
if col_num >= arr.shape[1]:
return arr
# sort array over given column
arr_sorted = arr[arr[:, col_num].argsort()]
# if the number of 1s in the given column is not equal to the total number
# of rows neither equal to 0, split on 1 and 0, sort and then merge
if len(arr) > np.sum(arr_sorted[:, col_num]) > 0:
arr_sorted0s = sort_col(arr_sorted[arr_sorted[:, col_num]==0], col_num+1)
arr_sorted1s = sort_col(arr_sorted[arr_sorted[:, col_num]==1], col_num+1)
# change order of stacking if you want ascenting order
return np.vstack((arr_sorted0s, arr_sorted1s))
# if the number of 1s in the given column is equal to the total number
# of rows or equal to 0, just go to the next iteration
return sort_col(arr_sorted, col_num + 1)
np.random.seed(0)
a = np.random.randint(0, 2, (5, 4))
print(a)
print(sort_col(a))
# prints
[[0 1 1 0]
[1 1 1 1]
[1 1 1 0]
[0 1 0 0]
[0 0 0 1]]
[[0 0 0 1]
[0 1 0 0]
[0 1 1 0]
[1 1 1 0]
[1 1 1 1]]
Edit. Or better yet use Daniels solution. I didn't check for new answers before I posted my code.

How to cluster dataset with more than 3 D and interet/Visualize?

Lets give an example
X: 1 2 3 4 5
Y: .9 .91 .92 .93 .94
Z: 20 36 999 211
M. 4000 3456 1 0
When I have such dataset, Which clustering algorithm to choose ? Also, How to interpret the results after clustering ?
Meaning: How to feed 4D dataset into cluster.
I found DBSCAN available on internet for 2D with which plot is possible. Since my dataset is 4 D and varies ILLOGICALLY...I am afraid to feed this to Algorithm
`
import pdb
import matplotlib.pyplot as plt
from numpy.random import rand
from numpy import square, sqrt
def regionQuery(P, eps, D):
neighbourPts = []
for point in D:
#print point
if sqrt(square(P[1] - point[1]) + square(P[2] - point[2]))<eps:
neighbourPts.append(point)
return neighbourPts
def DBSCAN(D, eps, MinPts):
noise = []
visited = []
C = []
c_n = -1
for point in D:
visited.append(point) #marking point as visited
# print point
neighbourPts = regionQuery(point, eps, D)
if len(neighbourPts) < MinPts:
noise.append(point)
else:
C.append([])
c_n+=1
expandCluster(point, neighbourPts, C, c_n,eps, MinPts, D, visited)
print("no. of clusters: " , len(C) )
print("length of noise:", len(noise))
for cluster in C:
col =[rand(1),rand(1),rand(1)]
#print(cluster)
plt.scatter([i[1] for i in cluster],[i[2] for i in cluster],color=col)
plt.show()
def expandCluster(P, neighbourPts, C, c_n,eps, MinPts, D, visited):
C[c_n].append(P)
for point in neighbourPts:
if point not in visited:
visited.append(point)
neighbourPts_2 = regionQuery(point, eps, D)
if len(neighbourPts_2) >= MinPts:
neighbourPts += neighbourPts_2
if point not in (i for i in C):
C[c_n].append(point)
eps =20#input("enter eps")
x=200*rand(10)
y=200*rand(10)
l=[]
for i in range(10):
l.append([i,x[i],y[i]])
#pdb.set_trace()
DBSCAN(l,eps,1)`

If you are using Python:
Approach 1:
from sklearn.metrics import confusion_matrix as cm
import pandas as pd
y_test = [1, 0, 0]
y_pred = [1, 0, 0]
confusion_matrix=cm(y_test, y_pred)
list1 = ["Actual 0", "Actual 1"]
list2 = ["Predicted 0", "Predicted 1"]
pd.DataFrame(confusion_matrix, list1,list2)
Approach 2:
While sklearn.metrics.confusion_matrix provides a numeric matrix, you can generate a 'report' using the following:
import pandas as pd
y_true = pd.Series([2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2])
y_pred = pd.Series([0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2])
pd.crosstab(y_true, y_pred, rownames=['True'], colnames=['Predicted'], margins=True)
which results in:
Predicted 0 1 2 All
True
0 3 0 0 3
1 0 1 2 3
2 2 1 3 6
All 5 2 5 12
This allows us to see that:
The diagonal elements show the number of correct classifications for each class: 3, 1 and 3 for the classes 0, 1 and 2.
The off-diagonal elements provides the misclassifications: for example, 2 of the class 2 were misclassified as 0, none of the class 0 were misclassified as 2, etc.
The total number of classifications for each class in both y_true and y_pred, from the "All" subtotals
This method also works for text labels, and for a large number of samples in the dataset can be extended to provide percentage reports.

2D binary list python

I'm having trouble making a function that places a number inside a binary grid. For instance,
if i'm given 4 3 2 1, and I have a grid that is 5x5, it would look like the following...
4 4 4 4 1
4 4 4 4 0
4 4 4 4 0
4 4 4 4 0
0 0 0 0 0
My current code reads a text file and creates a list that is arranged in descending order. For instance if the text file contained 1 2 3, it would create a list of integers 3 2 1. Also my code prompts for a bin # which creates a binxbin square. I don't know how to actually place in a number 4 for the bin. This is the function that should place in the values which i'm stuck with.
def isSpaceFree(bin, row, column, block):
if row + block > len(bin):
return False
if column + block > len(bin):
return False
if bin[row][column] == 0 :
return True
else:
return False
for r in range(row, row+block):
if bin[row][column] != 0:

It sounds like isSpaceFree should return True if you can create a square with origin origin (row, column) and size block, without going out of bounds or overlapping any non-zero elements. In which case, you're 75% of the way there. You have the bounds checking ready, and half of the overlap check loop.
def isSpaceFree(bin, row, column, block):
#return False if the block would go out of bounds
if row + block > len(bin):
return False
if column + block > len(bin):
return False
#possible todo:
#return False if row or column is negative
#return False if the square would overlap an existing element
for r in range(row, row+block):
for c in range(column, column+block):
if bin[r][c] != 0: #oops, overlap will occur
return False
#square is in bounds, and doesn't overlap anything. Good to go!
return True
Then, actually placing the block is the same double-nested loop, but instead performing an assignment.
def place(bin, row, column, block):
if isSpaceFree(bin, row, column, block):
for r in range(row, row+block):
for c in range(column, column+block):
bin[r][c] = block
x = [
[0,0,0,0,0],
[0,0,0,0,0],
[0,0,0,0,0],
[0,0,0,0,0],
[0,0,0,0,0],
]
place(x, 0, 0, 4)
print "\n".join(str(row) for row in x)
Result:
[4, 4, 4, 4, 0]
[4, 4, 4, 4, 0]
[4, 4, 4, 4, 0]
[4, 4, 4, 4, 0]
[0, 0, 0, 0, 0]