I have two numpy arrays that are OpenCV convex hulls and I want to check for intersection without creating for loops or creating images and performing numpy.bitwise_and on them, both of which are quite slow in Python. The arrays look like this:
[[[x1 y1]]
[[x2 y2]]
[[x3 y3]]
...
[[xn yn]]]
Considering [[x1 y1]] as one single element, I want to perform intersection between two numpy ndarrays. How can I do that? I have found a few questions of similar nature, but I could not figure out the solution to this from there.
You can use a view of the array as a single dimension to the intersect1d function like this:
def multidim_intersect(arr1, arr2):
arr1_view = arr1.view([('',arr1.dtype)]*arr1.shape[1])
arr2_view = arr2.view([('',arr2.dtype)]*arr2.shape[1])
intersected = numpy.intersect1d(arr1_view, arr2_view)
return intersected.view(arr1.dtype).reshape(-1, arr1.shape[1])
This creates a view of each array, changing each row to a tuple of values. It then performs the intersection, and changes the result back to the original format. Here's an example of using it:
test_arr1 = numpy.array([[0, 2],
[1, 3],
[4, 5],
[0, 2]])
test_arr2 = numpy.array([[1, 2],
[0, 2],
[3, 1],
[1, 3]])
print multidim_intersect(test_arr1, test_arr2)
This prints:
[[0 2]
[1 3]]
you can use http://pypi.python.org/pypi/Polygon/2.0.4, here is an example:
>>> import Polygon
>>> a = Polygon.Polygon([(0,0),(1,0),(0,1)])
>>> b = Polygon.Polygon([(0.3,0.3), (0.3, 0.6), (0.6, 0.3)])
>>> a & b
Polygon:
<0:Contour: [0:0.60, 0.30] [1:0.30, 0.30] [2:0.30, 0.60]>
To convert the result of cv2.findContours to Polygon point format, you can:
points1 = contours[0].reshape(-1,2)
This will convert the shape from (N, 1, 2) to (N, 2)
Following is a full example:
import Polygon
import cv2
import numpy as np
from scipy.misc import bytescale
y, x = np.ogrid[-2:2:100j, -2:2:100j]
f1 = bytescale(np.exp(-x**2 - y**2), low=0, high=255)
f2 = bytescale(np.exp(-(x+1)**2 - y**2), low=0, high=255)
c1, hierarchy = cv2.findContours((f1>120).astype(np.uint8),
cv2.cv.CV_RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE)
c2, hierarchy = cv2.findContours((f2>120).astype(np.uint8),
cv2.cv.CV_RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE)
points1 = c1[0].reshape(-1,2) # convert shape (n, 1, 2) to (n, 2)
points2 = c2[0].reshape(-1,2)
import pylab as pl
poly1 = pl.Polygon(points1, color="blue", alpha=0.5)
poly2 = pl.Polygon(points2, color="red", alpha=0.5)
pl.figure(figsize=(8,3))
ax = pl.subplot(121)
ax.add_artist(poly1)
ax.add_artist(poly2)
pl.xlim(0, 100)
pl.ylim(0, 100)
a = Polygon.Polygon(points1)
b = Polygon.Polygon(points2)
intersect = a&b # calculate the intersect polygon
poly3 = pl.Polygon(intersect[0], color="green") # intersect[0] are the points of the polygon
ax = pl.subplot(122)
ax.add_artist(poly3)
pl.xlim(0, 100)
pl.ylim(0, 100)
pl.show()
Output:
So this is what I did to get the job done:
import Polygon, numpy
# Here I extracted and combined some contours and created a convex hull from it.
# Now I wanna check whether a contour acquired differently intersects with this hull or not.
for contour in contours: # The result of cv2.findContours is a list of contours
contour1 = contour.flatten()
contour1 = numpy.reshape(contour1, (int(contour1.shape[0]/2),-1))
poly1 = Polygon.Polygon(contour1)
hull = hull.flatten() # This is the hull is previously constructued
hull = numpy.reshape(hull, (int(hull.shape[0]/2),-1))
poly2 = Polygon.Polygon(hull)
if (poly1 & poly2).area()<= some_max_val:
some_operations
I had to use for loop, and this altogether looks a bit tedious, although it gives me expected results. Any better methods would be greatly appreciated!
inspired by jiterrace's answer
I came across this post while working with Udacity deep learning class(trying to find the overlap between training and test data).
I am not familiar with "view" and found the syntax a bit hard to understand, probably the same when I try to communicate to my friends who think in "table".
My approach is basically to flatten/reshape the ndarray of shape (N, X, Y) into shape (N, X*Y, 1).
print(train_dataset.shape)
print(test_dataset.shape)
#(200000L, 28L, 28L)
#(10000L, 28L, 28L)
1). INNER JOIN (easier to understand, slow)
import pandas as pd
%%timeit -n 1 -r 1
def multidim_intersect_df(arr1, arr2):
p1 = pd.DataFrame([r.flatten() for r in arr1]).drop_duplicates()
p2 = pd.DataFrame([r.flatten() for r in arr2]).drop_duplicates()
res = p1.merge(p2)
return res
inters_df = multidim_intersect_df(train_dataset, test_dataset)
print(inters_df.shape)
#(1153, 784)
#1 loop, best of 1: 2min 56s per loop
2). SET INTERSECTION (fast)
%%timeit -n 1 -r 1
def multidim_intersect(arr1, arr2):
arr1_new = arr1.reshape((-1, arr1.shape[1]*arr1.shape[2])) # -1 means row counts are inferred from other dimensions
arr2_new = arr2.reshape((-1, arr2.shape[1]*arr2.shape[2]))
intersected = set(map(tuple, arr1_new)).intersection(set(map(tuple, arr2_new))) # list is not hashable, go tuple
return list(intersected) # in shape of (N, 28*28)
inters = multidim_intersect(train_dataset, test_dataset)
print(len(inters))
# 1153
#1 loop, best of 1: 34.6 s per loop
Related
I have a very large 3d numpy from which I want to extract many values (x, y, z).
For the sake of simplicity let's say this is the numpy:
import numpy as np
a = np.arange(64).reshape(4,4,4)
From which I want to extract the values of the following collection of points:
points = [[3,0,0],[0,1,0],[3,0,1],[2,3,1]]
In this example, the expected result should be:
[48,4,49,45]
Because performance metter, I want to avoid iterate like the following code:
points = [[3,0,0],[0,1,0],[3,0,1],[2,3,1]]
for i in points:
print(a[i[0],i[1],i[2]])
Try this. Uses numpy fancy/advanced indexing.
>>> import numpy as np
>>> a = np.arange(64).reshape(4,4,4)
>>> points = [[3,0,0],[0,1,0],[3,0,1],[2,3,1]]
>>> points = np.array(points)
>>> i = points[:, 0]
>>> j = points[:, 1]
>>> k = points[:, 2]
>>> a[i, j, k]
array([48, 4, 49, 45])
I want to implement a "row wise" matrix multiplication.
More specifically speaking, I want to plot a set of arrows whose directions range from (-pi, pi). The following code is how I implemented it.
scan_phi = np.linspace(-np.pi*0.5, np.pi*0.5, 450)
points = np.ones((450, 2), dtype=np.float)
points[..., 0] = 0.0
n_pts = len(points)
sin = np.sin(scan_phi)
cos = np.cos(scan_phi)
rot = np.append(np.expand_dims(np.vstack([cos, -sin]).T, axis=1),
np.expand_dims(np.vstack([sin, cos]).T, axis=1),
axis=1)
points_rot = []
for idx, p in enumerate(points):
points_rot.append(np.matmul(rot[idx], p.T))
points_rot = np.array(points_rot)
sample = points_rot[::10]
ax = plt.axes()
ax.set_xlim(-2, 2)
ax.set_ylim(-2, 2)
for idx, p in enumerate(sample):
if idx == 0:
ax.arrow(0, 0, p[0], p[1], head_width=0.05, head_length=0.1, color='red')
else:
ax.arrow(0, 0, p[0], p[1], head_width=0.05, head_length=0.1, fc='k', ec='k')
plt.show()
In my code, "rot" ends up being an array of (450, 2, 2) meaning for each arrow, I have created a corresponding rotation matrix to rotate it. I have 450 points stored in "points" (450, 2) that I want to draw arrows with. (Here the arrows are all initialized with [0, 1]. However, it can be initialized with different values which is why I want to have 450 individual points instead of just rotating a single point by 450 different angles)
The way I did is using a for-loop, i.e. for each arrow, I transform it individually.
points_rot = []
for idx, p in enumerate(points):
points_rot.append(np.matmul(rot[idx], p.T))
points_rot = np.array(points_rot)
However, I wonder if there's any nicer and easy way to do this completely through numpy, such as some operations that can perform matrix multiplication row-wise. Any idea will be grateful, thanks in advance!
This is a nice use-case for np.einsum:
aa = np.random.normal(size=(450, 2, 2))
bb = np.random.normal(size=(450, 2))
cc = np.einsum('ijk,ik->ij', aa, bb)
So that each row of cc is the product of corresponding rows of aa and bb:
np.allclose(aa[3].dot(bb[3]), cc) # returns True
Explanation: the Einstein notation ijk,ik->ij is saying:
cc[i,j] = sum(aa[i,j,k] * bb[i,k] for k in range(2))
I.e., all variables that do not appear in the right-hand side are summed away.
I have the following problem. I have 3D point coordinates and I want to conenct them in an array or "draw a line" like you can do it in 2D with skimage(http://scikit-image.org/docs/0.13.x/api/skimage.draw.html#skimage.draw.line). The optimal case would be if i could directly draw a cylinder with a radius in an array and give different radius different values like rings arround the line.(wrinkles could be a problem here). There have been approaches to do that but they are not the right thing i think like here:
"Draw" a 3d line into an array
With the two approaches of #Paul Panzer (Fastest way to get all the points between two (X,Y) coordinates in python) you get all coordinates between two 2D Points, but how would it look like in 3D, esspecially the second approach which is faster?:
import numpy as np
from timeit import timeit
def connect(ends):
d0, d1 = np.abs(np.diff(ends, axis=0))[0]
if d0 > d1:
return np.c_[np.linspace(ends[0, 0], ends[1, 0], d0+1, dtype=np.int32),
np.linspace(ends[0, 1]+0.5, ends[1, 1]+0.5, d0+1, dtype=np.int32)]
else:
return np.c_[np.linspace(ends[0, 0]+0.5, ends[1, 0]+0.5, d1+1, dtype=np.int32),
np.linspace(ends[0, 1], ends[1, 1], d1+1, dtype=np.int32)]
def connect2(ends):
d0, d1 = np.diff(ends, axis=0)[0]
if np.abs(d0) > np.abs(d1):
return np.c_[np.arange(ends[0, 0], ends[1,0] + np.sign(d0), np.sign(d0), dtype=np.int32),
np.arange(ends[0, 1] * np.abs(d0) + np.abs(d0)//2,
ends[0, 1] * np.abs(d0) + np.abs(d0)//2 + (np.abs(d0)+1) * d1, d1, dtype=np.int32) // np.abs(d0)]
else:
return np.c_[np.arange(ends[0, 0] * np.abs(d1) + np.abs(d1)//2,
ends[0, 0] * np.abs(d1) + np.abs(d1)//2 + (np.abs(d1)+1) * d0, d0, dtype=np.int32) // np.abs(d1),
np.arange(ends[0, 1], ends[1,1] + np.sign(d1), np.sign(d1), dtype=np.int32)]
ends = np.array([[ 1520, -1140],
[ 1412, -973]])
Since you tagged the question 'vtk', I assume you can use it in your code. In that case, I think vtkTubeFilter does exactly what you are looking for - you create a vtkPolyData with all your line segments (you have your endpoints for that) and for each segment, the filter generates a cylinder. You can also specify different radius for each segment as you wanted: call SetVaryRadiusToVaryRadiusByScalar() on the filter to turn it on and provide the array with radii as scalar data.
You can find a code example that does all that (and more) here: https://www.vtk.org/Wiki/VTK/Examples/Cxx/VisualizationAlgorithms/TubesWithVaryingRadiusAndColors
This is my first nontrivial use of numpy, and I'm having some trouble in one spot.
So, I have colors, a (xsize + 2, ysize + 2, 3) ndarray, and newlife, a (xsize + 2, ysize + 2) ndarray of booleans. I want to add a random value between -5 and 5 to all three values in colors at all positions where newlife is true. In other words newlife maps 2D vectors to whether or not I want to add a random value to the color in colors at that position.
I've tried a million variations on this:
colors[np.nonzero(newlife)] += (np.random.random_sample((xsize + 2,ysize + 2, 3)) * 10 - 5)
but I keep getting stuff like
ValueError: operands could not be broadcast together with shapes (589,3) (130,42,3) (589,3)
How do I do this?
I think this does what you want:
# example data
colors = np.random.randint(0, 100, (5,4,3))
newlife = np.random.randint(0, 2, (5,4), bool)
# create values to add, then mask with newlife
to_add = np.random.randint(-5,6, (5,4,3))
to_add[~newlife] = 0
# modify in place
colors += to_add
This changes the colors in-place assuming uint8 dtype. Both assumptions are not essential:
import numpy as np
n_x, n_y = 2, 2
colors = np.random.randint(5, 251, (n_x+2, n_y+2, 3), dtype=np.uint8)
mask = np.random.randint(0, 2, (n_x+2, n_y+2), dtype=bool)
n_change = np.count_nonzero(mask)
print(colors)
print(mask)
colors[mask] += np.random.randint(-5, 6, (n_change, 3), dtype=np.int8).view(np.uint8)
print(colors)
The easiest way of understanding this is to look at the shape of colors[mask].
I am having a very difficult time vectoring, I can't seem to think about math in that way yet. I have this right now:
#!/usr/bin/env python
import numpy as np
import math
grid = np.zeros((2,2))
aList = np.arange(1,5).reshape(2,2)
i,j = np.indices((2,2))
iArray = (i - aList[:,0:1])
jArray = (j - aList[:,1:2])
print np.power(np.power(iArray, 2) + np.power(jArray, 2), .5)
My print out looks like this:
[[ 2.23606798 1.41421356]
[ 4.47213595 3.60555128]]
What I am trying to do is take a 2D array of pixel values, grid, and say how far each pixel is from a list of important pixels, aList.
# # #
# # #
* # *
An example is if the *s (0,2) and (2,2) are important pixels and I am currently at the # (2,0) pixel, my value for the # pixel would be:
[(0-2)^2 + (2-0)^2]^.5 + [(2-2)^2 + (0-2)^2]^.5
All grid does is hold pixel values so I need to get the index of each pixel value to associate distance. However my Alist array holds [x,y] coordinates, So that one is easy. I think I right now I have two issues:
1. I am not getting the indeces correctly
2. I am not looping over the coordinates in aList properly
With a little help from broadcasting, I get this, with data based on your last example:
import numpy as np
grid = np.zeros((3, 3))
aList = np.array([[2, 0], [2, 2]])
important_rows, important_cols = aList.T
rows, cols = np.indices(grid.shape)
dist = np.sqrt((important_rows - rows.ravel()[:, None])**2 +
(important_cols - cols.ravel()[:, None])**2).sum(axis=-1)
dist = dist.reshape(grid.shape)
>>> dist
array([[ 4.82842712, 4.47213595, 4.82842712],
[ 3.23606798, 2.82842712, 3.23606798],
[ 2. , 2. , 2. ]])
You can get more memory efficient by doing:
important_rows, important_cols = aList.T
rows, cols = np.meshgrid(np.arange(grid.shape[0]),
np.arange(grid.shape[1]),
sparse=True, indexing='ij')
dist2 = np.sqrt((rows[..., None] - important_rows)**2 +
(cols[..., None] - important_cols)**2).sum(axis=-1)
My approach:
import numpy as np
n = 3
aList = np.zeros([n,n])
distance = np.zeros([n,n])
I,J = np.indices([n,n])
aList[2,2] = 1; aList[0,2] = 1 #Importan pixels
important = np.where(aList == 1) #Where the important pixels are
for i,j in zip(I[important],J[important]): #This part could be improved...
distance += np.sqrt((i-I)**2+(j-J)**2)
print distance
The last 'for' could be improved, but if you have only a few important pixels, the performance will be good...
Checking with:
import matplotlib.pyplot as plt
n = 500
...
aList[249+100,349] = 1; aList[249-100,349] = 1 ;aList[249,50] = 1
...
plt.plot(I[important],J[important],'rx',markersize=20)
plt.imshow(distance.T,origin='lower',
cmap=plt.cm.gray)
plt.show()
The result is very comfortable: