Related
I am trying to create a big array for a high dim in y_shift = np.fromfunction(lambda i,j: (i)>>j, ((2**dim), dim), dtype=np.uint32). For example dim=32. I have two questions
1.- How to improve the speed in term of time
2.- How to avoid the message for dim=32 zsh: killed python3
EDIT::
Alternative you can consider to use uint8 instead of uint32
y_shift = np.fromfunction(lambda i,j: (1&(i)>>j), ((2**dim), dim), dtype=np.uint8)
To answer your question:
You get the error zsh: killed python3 because you run out of memory.
If you want to run the code you initially proposed:
dim =32
y_shift = np.fromfunction(lambda i,j: (i)>>j, ((2**dim), dim), dtype=np.uint32)
You would need more than 500GB of memory, see here.
I would recommend thinking of alternatives and avoid trying to save the entire array to memory.
fromfunction just does 2 things:
args = indices(shape, dtype=dtype)
return function(*args, **kwargs)
It makes the indices array
In [247]: args = np.indices(((2**4),4))
In [248]: args.shape
Out[248]: (2, 16, 4)
and it passes that array to your function
In [249]: args[0]>>args[1]
Out[249]:
array([[ 0, 0, 0, 0],
[ 1, 0, 0, 0],
[ 2, 1, 0, 0],
[ 3, 1, 0, 0],
...
[14, 7, 3, 1],
[15, 7, 3, 1]])
With dim=32:
In [250]: ((2**32),32)
Out[250]: (4294967296, 32)
the resulting args array will be (2, 4294967296, 32). There's no way around that in terms of speed or memory use.
I would like to know if there is an efficient method to get sub-arrays from a larger numpy array.
What I have is an application of np.where. I iterate 'manually' over x and y as offsets and apply where with a kernel to each rectangle extracted from the larger array with proper dimensions.
But is there a more direct approach in numpy's collection of methods?
import numpy as np
example = np.arange(20).reshape((5, 4))
# e.g. a cross kernel
a_kernel = np.asarray([[0, 1, 0], [1, 1, 1], [0, 1, 0]])
np.where(a_kernel, example[1:4, 1:4], 0)
# returns
# array([[ 0, 6, 0],
# [ 9, 10, 11],
# [ 0, 14, 0]])
def arrays_from_kernel(a, a_kernel):
width, height = a_kernel.shape
y_max, x_max = a.shape
return [np.where(a_kernel, a[y:(y + height), x:(x + width)], 0)
for y in range(y_max - height + 1)
for x in range(x_max - width + 1)]
sub_arrays = arrays_from_kernel(example, a_kernel)
This returns the arrays I need for further processing.
# [array([[0, 1, 0],
# [4, 5, 6],
# [0, 9, 0]]),
# array([[ 0, 2, 0],
# [ 5, 6, 7],
# [ 0, 10, 0]]),
# ...
# array([[ 0, 9, 0],
# [12, 13, 14],
# [ 0, 17, 0]]),
# array([[ 0, 10, 0],
# [13, 14, 15],
# [ 0, 18, 0]])]
The context: similar to 2D convolution I would like to apply a custom function on each of the subarrays (e.g. product of squared numbers).
At the moment, you're manually advancing a sliding window over the data - stride tricks to the rescue! (And no, I didn't just make that up - there's actually a submodule called stride_tricks in numpy!) Instead of manually building windows into the data, and calling np.where() on them, if you had the windows in an array, you could call np.where() just once. Stride tricks allow you to create such an array without even having to copy the data.
Let me explain. Normal slices in numpy create views into the original data instead of copies. This is done by referring to the original data, but changing the strides used to access the data (ie. how much to jump between two elements or two rows, and so on). Stride tricks allow you to modify those strides more freely than just slicing and reshaping does, so you can eg. iterate over the same data more than once, which is useful here.
Let me demonstrate:
import numpy as np
example = np.arange(20).reshape((5, 4))
a_kernel = np.array([[0, 1, 0], [1, 1, 1], [0, 1, 0]])
def sliding_window(data, win_shape, **kwargs):
assert data.ndim == len(win_shape)
shape = tuple(dn - wn + 1 for dn, wn in zip(data.shape, win_shape)) + win_shape
strides = data.strides * 2
return np.lib.stride_tricks.as_strided(data, shape=shape, strides=strides, **kwargs)
def arrays_from_kernel(a, a_kernel):
windows = sliding_window(a, a_kernel.shape)
return np.where(a_kernel, windows, 0)
sub_arrays = arrays_from_kernel(example, a_kernel)
The scipy.ndimage module offers a number of filters -- one of which might meet your needs. If none of those filters do what you want, you could use ndimage.generic_filter
to call a custom function on each subarray. ndimage.generic_filter is not as fast as the other ndimage filters, however.
For example,
import numpy as np
example = np.arange(20).reshape((5, 4))
a_kernel = np.asarray([[0, 1, 0], [1, 1, 1], [0, 1, 0]])
# def arrays_from_kernel(a, a_kernel):
# width, height = a_kernel.shape
# y_max, x_max = a.shape
# return [np.where(a_kernel, a[y:(y + height), x:(x + width)], 0)
# for y in range(y_max - height + 1)
# for x in range(x_max - width + 1)]
# sub_arrays = arrays_from_kernel(example, a_kernel)
# for arr in sub_arrays:
# print(arr)
# print('-'*80)
import scipy.ndimage as ndimage
def func(x):
# reject subarrays that extend beyond the border of the `example` array
if not np.isnan(x).any():
y = np.zeros_like(a_kernel, dtype=example.dtype)
np.put(y, np.flatnonzero(a_kernel), x)
print(y)
# Instead or returning 0, you can perform your desired computation on the subarray here.
# Note that you may not need the 2D array y; often, you only need the values in the 1D array x
return 0
result = ndimage.generic_filter(example, func, footprint=a_kernel, mode='constant', cval=np.nan)
For the particular problem of computing the product of squares for each subarray, you
could convert the product into a sum by taking advantage of the fact that A * B = exp(log(A)+log(B)). This would allow you to express the computation as a normal convolution. Now using ndimage.convolve can improve performance a lot. The amount of the improvement depends on the size of example:
import numpy as np
import scipy.ndimage as ndimage
import perfplot
a_kernel = np.asarray([[0, 1, 0], [1, 1, 1], [0, 1, 0]])
def orig(example, a_kernel=a_kernel):
def arrays_from_kernel(a, a_kernel):
width, height = a_kernel.shape
y_max, x_max = a.shape
return [
np.where(a_kernel, a[y : (y + height), x : (x + width)], 1)
for y in range(y_max - height + 1)
for x in range(x_max - width + 1)
]
return [np.prod(x) ** 2 for x in arrays_from_kernel(example, a_kernel)]
def alt(example, a_kernel=a_kernel):
logged = np.log(example)
result = ndimage.convolve(logged, a_kernel, mode="constant", cval=0)[1:-1, 1:-1]
return (np.exp(result) ** 2).ravel()
def make_example(N):
return np.random.random(size=(N, N))
def check(A, B):
return np.allclose(A, B)
perfplot.show(
setup=make_example,
kernels=[orig, alt],
n_range=[2 ** k for k in range(2, 11)],
logx=True,
logy=True,
xlabel="len(example)",
equality_check=check,
)
I want to calculate the Perceived brightness ,based on the formula from this link, of this image:
The idea is to loop over each pixel and calculate its Perceived brightness according to this formula:
Pb = sqrt(0.241 R² + 0.691 G² + 0.068 B²)
then sum all the values and calculate the mean.
Here is the code I wrote:
import cv2
from math import sqrt
img = cv2.imread('e.png')
H, W = img.shape[:2]
pr = 0.241
pg = 0.691
pb = 0.068
p = []
for h in range(0, H):
for w in range(0, W):
p.append(sqrt(pr * pow(img[h][w][2], 2) + pg * pow(img[h][w][1], 2) + pb * pow(img[h][w][0], 2)))
arr = np.reshape(p, (H, W))
cv2.imwrite('loop_img.jpg', arr)
print(np.mean(arr))
The image I got at the end is this:
And the mean is 82.04557421656007
However when I repeated the same process using numpy (to avoid looping over each pixel), I got different values!
Here is the code I used:
import cv2
import numpy as np
img = cv2.imread('e.png')
b, g, r = cv2.split(img)
pr = 0.241
pg = 0.691
pb = 0.068
P = np.sqrt(pr * pow(r, 2) + pg * pow(g, 2) + pb * pow(b, 2))
cv2.imwrite('np_img.jpg', P)
print(np.mean(P))
The image I got is this:
And the mean is 1.6438602314083277
The most weird is that when I applied the same methods on a random numpy array, I got similar results!
import numpy as np
import cv2
from math import sqrt
pr = 0.241
pg = 0.691
pb = 0.068
arr = np.array([[[255, 127, 0],
[255, 127, 0]],
[[255, 133, 0],
[255, 133, 0]],
[[255, 138, 0],
[255, 138, 0]]])
b, g, r = cv2.split(arr)
p = []
for h in range(0, 3):
for w in range(0, 2):
print(arr[h][w])
p.append(sqrt(pr * pow(arr[h][w][2], 2) + pg * pow(arr[h][w][1], 2) + pb * pow(arr[h][w][0], 2)))
arr_p = np.reshape(p, (3, 2))
print('arr_p:', arr_p)
np_p = np.sqrt(pr * pow(r, 2) + pg * pow(g, 2) + pb * pow(b, 2))
print('np_ap:', np_p)
print('loop_mean:', np.mean(arr_p))
print('numpy_mean:', np.mean(np_p))
The results I got:
arr_p: [[124.7671391 124.7671391 ]
[129.01472397 129.01472397]
[132.59375551 132.59375551]]
np_ap: [[124.7671391 124.7671391 ]
[129.01472397 129.01472397]
[132.59375551 132.59375551]]
loop_mean: 128.79187285939827
numpy_mean: 128.79187285939827
Is there any explanation why I got different results with the image and similar results with the second array? (could it be related to the array elements type?)
N.B: I use
python==3.6
numpy==1.16.1
opencv-contrib-python==4.0.0.21
opencv-python==4.0.0.21
The problem is due to the difference of data type conversion rules between numpy array and raw data types.
In the case of numpy array, the calculation is being done as follows:
P = np.sqrt(pr * pow(r, 2) + pg * pow(g, 2) + pb * pow(b, 2))
The culprit operation here is pow. Since the default data type of image read using cv2.imread is np.uint8 so consequently, r, g and b also have the same type. Now, when pow function is applied on the numpy array, the resultant array tends to have the same integer type. The values in the result are truncated to the range of uint8 type thus causing invalid results. Since the results are truncated, the mean value becomes very small as being observed.
Possible solutions:
1. Convert input image to floating point type:
img = cv2.imread('e.png')
img = img.astype(np.float)
2. Use floating point operands in pow:
P = np.sqrt(pr * pow(r, 2.0) + pg * pow(g, 2.0) + pb * pow(b, 2.0))
Why are the results correct in the loop case?
p.append(sqrt(pr * pow(img[h][w][2], 2) + pg * pow(img[h][w][1], 2) + pb * pow(img[h][w][0], 2)))
Apparently, applying pow on a single integer instead of numpy array results in a value of larger integer type (int64) thus avoiding the issue of truncation.
The problem is the pow function to a np.uint8 array. First, lets have a simple example:
>> a = np.arange(20, dtype=np.uint8).reshape(4,5)
which gives:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]], dtype=uint8)
It is important to test with np.uint8 which is the type of a loaded image. Then we do pow or np.power (they behave exactly the same) and the result is the following:
>> np.power(a,2)
array([[ 0, 1, 4, 9, 16],
[ 25, 36, 49, 64, 81],
[100, 121, 144, 169, 196],
[225, 0, 33, 68, 105]], dtype=uint8)
>> pow(a,2)
array([[ 0, 1, 4, 9, 16],
[ 25, 36, 49, 64, 81],
[100, 121, 144, 169, 196],
[225, 0, 33, 68, 105]], dtype=uint8)
As you can see, the power function did not change the type... and this leads to overflow...
You have 2 options to solve it:
Cast the type and then cast it back like
b = np.float32(b) #same for g and r or to the whole image
# or this
b, g, r = cv2.split(np.float32(img))
and then before saving use np.uint8(), opencv saving functions usually work only with uint8... maybe the newer versions doesn't.
The other thing is to use np.float_power which will return float32 type and the correct numbers.
This code works as expected:
import numpy as np
from PIL import Image, ImageDraw
A = (
( 2, 2),
( 2, 302),
( 302, 302),
( 302, 2)
)
img = Image.new('L', (310, 310), 0)
ImageDraw.Draw(img).polygon(A, outline=1, fill=1)
mask = np.array(img)
print(mask)
However, if the A matrix is provided as numpy array:
A = np.array(
[[ 2, 2],
[ 2, 302],
[302, 302],
[302, 2]], dtype="int32"
)
it produces completely wrong result. I also try to flatten the A array, it does not help.
Do I miss something? Can I stuff the numpy array somehow directly into PIL?
If call-interaface says use a list-of-tuples or a list of interleaved values,
best use a list-of-tuples or a sequence / list of interleaved values:
PIL.ImageDraw.ImageDraw.polygon( xy, fill = None, outline = None )
Draws a polygon.
The polygon outline consists of straight lines between the given coordinates, plus a straight line between the last and the first coordinate.
xy – Sequence of either 2-tuples like [(x, y), (x, y), ...]ornumeric values like [x, y, x, y, ...].
Can I stuff ..
Using
>>> xy
array([[ 2, 3],
[10, 3],
[10, 0],
[ 2, 0]])
>>> xy.flatten().tolist()
[ 2, 3, 10, 3, 10, 0, 2, 0 ]
>>>
shall work and meet the PIL-documented-Call-Interface for ImageDraw.polygon()
I want to rotate an array but not as a whole, only small portion of it.
I have 512X512 array (basically it is a Gaussian circle at the center (150,150) with 200 radius). Now I want to rotate only small portion (center around (150,150) with radius 100) of the array by 90 degree. Initially I used numpy rot90 module but it rotate each array element which is not I want.
If you can describe the elements that you would like rotated using advanced indexing, then you should be able to perform the rotation using something like the following (assuming your array is called arr):
arr[rs:re,cs:ce] = np.rot90(np.copy(arr[rs:re,cs:ce]))
Here rs, re, cs, and ce would signify the row-start and row-end of a slice, and the column-start and column-end of a slice, respectively.
Here is an example of why the np.copy call is necessary (at least in numpy 1.3.0):
>>> import numpy as np
>>> m = np.array([[i]*4 for i in range(4)])
>>> m
array([[0, 0, 0, 0],
[1, 1, 1, 1],
[2, 2, 2, 2],
[3, 3, 3, 3]])
>>> m[1:3,1:3] = np.rot90(m[1:3,1:3]) # rotate middle 2x2
>>> m
array([[0, 0, 0, 0],
[1, 1, 2, 1], # got 1, 2 expected 1, 2
[2, 1, 1, 2], # 1, 1 1, 2
[3, 3, 3, 3]])
Here is some fuller code that does as F.J. has already explained.
And here is the code:
import numpy as np
import scipy
def circle(im, centre_x, centre_y, radius):
grid_x, grid_y = np.mgrid[0:im.shape[0],0:im.shape[1]]
return (grid_x-centre_x)**2 + (grid_y-centre_y)**2 < radius**2
centre_x, centre_y, radius = 150, 200, 100
x_slice = slice(centre_x - radius, centre_x + radius)
y_slice = slice(centre_y - radius, centre_y + radius)
im = scipy.misc.imread('1_tree.jpg')
rotated_square = np.rot90(im[x_slice,y_slice].copy())
im[circle(im, centre_x, centre_y,radius)] = rotated_square[circle(rotated_square,
radius, radius, radius)]
scipy.misc.imsave('sdffs.png',im)