Related
Suppose I have a 2D array with shape (3, 3), call it a, and an array of zeros with shape (7, 7, 5, 5), call it b. I want to modify b in the following way:
for p in range(5):
for q in range(5):
b[p:p + 3, q:q + 3, p, q] = a
Given:
a = np.array([[4, 2, 2],
[9, 0, 5],
[9, 9, 4]])
b = np.zeros((7, 7, 5, 5), dtype=int)
b would end up something like:
>>> b[:, :, 0, 0]
array([[4, 2, 2, 0, 0, 0, 0],
[9, 0, 5, 0, 0, 0, 0],
[9, 9, 4, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]])
>>> b[:, :, 0, 1]
array([[0, 4, 2, 2, 0, 0, 0],
[0, 9, 0, 5, 0, 0, 0],
[0, 9, 9, 4, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]])
One way to think about this to make a sliding window view of b (6D), slice out the parts you want (3D or 4D), and assign a to them.
However, there is a simpler way to do this altogether. The way a sliding window view works is by creating a dimension that steps along less than the full size of the dimension you are viewing. For example:
>>> x = np.array([1, 2, 3, 4])
array([1, 2, 3, 4])
>>> window = np.lib.stride_tricks.as_strided(
x, shape=(x.shape[0] - 2, 3),
strides=x.strides * 2)
[[1 2 3]
[2 3 4]]
I'm deliberately using np.lib.stride_tricks.as_strided rather than np.lib.stride_tricks.sliding_window_view here because it has a certain flexibility that you need.
You can have a stride that is larger than the axis you are viewing, as long as you are careful. Contiguous arrays are more forgiving in this case, but by no means a requirement. An example of this is np.diag. You can implement it something like this:
>>> x = np.arange(12).reshape(3, 4)
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
>>> diag = np.lib.stride_tricks.as_strided(
x, shape=(min(x.shape),),
strides=(sum(x.strides),))
array([ 0, 5, 10])
The trick is to make a view of only the parts of b you care about in a way that makes the assignment easy. Because of broadcasting rules, you will want the last two dimensions of the view to be a.shape, and the strides to be b.strides[:2], since that's where you want to place a.
The first two dimensions of the view will be responsible for making the copies of a. You want 25 copies, so the shape will be (5, 5). The strides are the trickier part. Let's take a look at a 2D case, just because that's easier to visualize, and then attempt to generalize:
>>> a0 = np.array([1, 2])
>>> b0 = np.zeros((4, 3), dtype=int)
>>> b0[0:2, 0] = b0[1:3, 1] = b0[2:4, 2] = a0
The goal is to make a view that strides along the diagonal of b0 in the first axis. So:
>>> np.lib.stride_tricks.as_strided(
b0, shape=(b0.shape[0] - a0.shape[0] + 1, a0.shape[0]),
strides=(sum(b0.strides), b0.strides[0]))[:] = a0
>>> b0
array([[1, 0, 0],
[2, 1, 0],
[0, 2, 1],
[0, 0, 2]])
So that's what you do for b, but adding up every second dimension:
a = np.array([[4, 2, 2],
[9, 0, 5],
[9, 9, 4]])
b = np.zeros((7, 7, 5, 5), dtype=int)
vshape = (*np.subtract(b.shape[:a.ndim], a.shape) + 1,
*a.shape)
vstrides = (*np.add(b.strides[:a.ndim], b.strides[a.ndim:]),
*b.strides[:a.ndim])
np.lib.stride_tricks.as_strided(b, shape=vshape, strides=vstrides)[:] = a
TL;DR
def emplace_window(a, b):
vshape = (*np.subtract(b.shape[:a.ndim], a.shape) + 1, *a.shape)
vstrides = (*np.add(b.strides[:a.ndim], b.strides[a.ndim:]), *b.strides[:a.ndim])
np.lib.stride_tricks.as_strided(b, shape=vshape, strides=vstrides)[:] = a
I've phrased it this way, because now you can apply it to any number of dimensions. The only expectations is that 2 * a.ndim == b.ndim and that b.shape[a.ndim:] == b.shape[:a.ndim] - a.shape + 1.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I wrote a backtracking Sudoku solving algorithm in Python.
It solves a 2D array like this (zero means "empty field"):
[
[7, 0, 0, 0, 0, 9, 0, 0, 3],
[0, 9, 0, 1, 0, 0, 8, 0, 0],
[0, 1, 0, 0, 0, 7, 0, 0, 0],
[0, 3, 0, 4, 0, 0, 0, 8, 0],
[6, 0, 0, 0, 8, 0, 0, 0, 1],
[0, 7, 0, 0, 0, 2, 0, 3, 0],
[0, 0, 0, 5, 0, 0, 0, 1, 0],
[0, 0, 4, 0, 0, 3, 0, 9, 0],
[5, 0, 0, 7, 0, 0, 0, 0, 2],
]
like this:
[
[7, 5, 8, 2, 4, 9, 1, 6, 3],
[4, 9, 3, 1, 5, 6, 8, 2, 7],
[2, 1, 6, 8, 3, 7, 4, 5, 9],
[9, 3, 5, 4, 7, 1, 2, 8, 6],
[6, 4, 2, 3, 8, 5, 9, 7, 1],
[8, 7, 1, 9, 6, 2, 5, 3, 4],
[3, 2, 7, 5, 9, 4, 6, 1, 8],
[1, 8, 4, 6, 2, 3, 7, 9, 5],
[5, 6, 9, 7, 1, 8, 3, 4, 2]
]
But for "hard" Sudokus (where there are a lot of zeros at the beginning), it's quite slow. It takes the algorithm around 9 seconds to solve the Sudoku above. That's a lot better then what I startet with (90 seconds), but still slow.
I think that the "deepcopy" can somehow be improved/replaced (because it is executed 103.073 times in the example below), but my basic approaches were slower..
I heard of 0.01 second C/C++ solutions but I'm not sure if those are backtracking algorithms of some kind of mathematical solution...
This is my whole algorithm with 2 example Sudokus:
from copy import deepcopy
def is_sol_row(mat,row,val):
m = len(mat)
for i in range(m):
if mat[row][i] == val:
return False
return True
def is_sol_col(mat,col,val):
m = len(mat)
for i in range(m):
if mat[i][col] == val:
return False
return True
def is_sol_block(mat,row,col,val):
rainbow = [0,0,0,3,3,3,6,6,6]
i = rainbow[row]
j = rainbow[col]
elements = {
mat[i + 0][j + 0], mat[i + 1][j + 0], mat[i + 2][j + 0],
mat[i + 0][j + 1], mat[i + 1][j + 1], mat[i + 2][j + 1],
mat[i + 0][j + 2], mat[i + 1][j + 2], mat[i + 2][j + 2],
}
if val in elements:
return False
return True
def is_sol(mat,row,col,val):
return is_sol_row(mat,row,val) and is_sol_col(mat,col,val) and is_sol_block(mat,row,col,val)
def findAllZeroIndizes(mat):
m = len(mat)
indizes = []
for i in range(m):
for j in range(m):
if mat[i][j] == 0:
indizes.append((i,j))
return indizes
def sudoku(mat):
q = [(mat,0)]
zeroIndizes = findAllZeroIndizes(mat)
while q:
t,numSolvedIndizes = q.pop()
if numSolvedIndizes == len(zeroIndizes):
return t
else:
i,j = zeroIndizes[numSolvedIndizes]
for k in range(1,10):
if is_sol(t,i,j,k):
newt = deepcopy(t)
newt[i][j] = k
q.append((newt,numSolvedIndizes+1))
return False
mat = [
[7, 0, 0, 0, 0, 9, 0, 0, 3],
[0, 9, 0, 1, 0, 0, 8, 0, 0],
[0, 1, 0, 0, 0, 7, 0, 0, 0],
[0, 3, 0, 4, 0, 0, 0, 8, 0],
[6, 0, 0, 0, 8, 0, 0, 0, 1],
[0, 7, 0, 0, 0, 2, 0, 3, 0],
[0, 0, 0, 5, 0, 0, 0, 1, 0],
[0, 0, 4, 0, 0, 3, 0, 9, 0],
[5, 0, 0, 7, 0, 0, 0, 0, 2],
]
# mat = [
# [3, 0, 6, 5, 0, 8, 4, 0, 0],
# [5, 2, 0, 0, 0, 0, 0, 0, 0],
# [0, 8, 7, 0, 0, 0, 0, 3, 1],
# [0, 0, 3, 0, 1, 0, 0, 8, 0],
# [9, 0, 0, 8, 6, 3, 0, 0, 5],
# [0, 5, 0, 0, 9, 0, 6, 0, 0],
# [1, 3, 0, 0, 0, 0, 2, 5, 0],
# [0, 0, 0, 0, 0, 0, 0, 7, 4],
# [0, 0, 5, 2, 0, 6, 3, 0, 0]
# ]
print(sudoku(mat))
The largest time sink is that, for every open position, you try each of the nine digits, without learning anything about the attempts. Your test grid has 56 open grid locations, so anything you do is magnified through that lens. A little preprocessing will go a long way. For instance, make a list of available numbers in each row and column. Key that appropriately, and use that for your search instead of range(m).
Another technique is to apply simple algorithms to make trivial placements as they become available. For instance, you can quickly derive the 1 in the upper-left block, and the missing 7s in the left and middle columns of blocks. This alone cuts the solution time in half. Wherever you're down to a single choice for what number goes in a selected open square, or where a selected number can be placed in a particular row/col/block, then make that placement before you engage in exhaustive backtracking.
Given a label map of dimensions W X H where each element can take values from {0,..,K-1} I want to output a label tensor of dimensions K X W x H where each element in the K'th map is 1 only if the corresponding value in the labelmap was K. Currently my implementation uses two for loops and is very slow.
p_label = Labelmap with one channel
label = np.zeros((K,p_label.shape[0], p_label.shape[1]))
for i in xrange(p_label.shape[0]):
for j in xrange(p_label.shape[1]):
label[p_label[i,j],i,j] = 1
Is there a better way to do this operation in Numpy using broadcasting?
You can use the == operator with broadcasting.
For example,
In [19]: W = 5
In [20]: H = 8
In [21]: K = 10
Create a p_label for the example:
In [22]: p_label = np.random.randint(0, K, size=(W, H))
kvals is simply an array containing [0, 1, ..., K-1]:
In [23]: kvals = np.arange(K)
kvals.reshape(-1, 1, 1) converts kvals to an array with shape (K, 1, 1). This is compared using == to p_label. Broadcasting applies, so the result of the comparison has shape (K, W, H). It is a boolean array of the values that you want. .astype(int) converts the result to an integer array. (You can remove that if a boolean array would work for you.)
In [24]: label = (p_label == kvals.reshape(-1, 1, 1)).astype(int)
Here's the original p_label. Note, for example, the locations of the value 0:
In [25]: p_label
Out[25]:
array([[3, 3, 2, 6, 2, 2, 9, 3],
[1, 8, 1, 1, 4, 3, 7, 8],
[5, 9, 1, 0, 7, 2, 8, 0],
[1, 3, 5, 4, 6, 0, 9, 5],
[5, 7, 2, 0, 6, 4, 5, 3]])
label[0] is 1 in the positions where p_label is 0.
In [26]: label[0]
Out[26]:
array([[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 1, 0, 0, 0, 1],
[0, 0, 0, 0, 0, 1, 0, 0],
[0, 0, 0, 1, 0, 0, 0, 0]])
Label[p_label, np.arange(p_label.shape[0])[:,None], np.arange(p_label.shape[1])] = 1
The 3 index arrays broadcast against each other.
==============================
lmap = np.arange(12).reshape(3,4)
lbl = np.zeros((12,3,4),int)
lbl[lmap,np.arange(3)[:,None],np.arange(4)] = 1
In [5]: lbl
Out[5]:
array([[[1, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0]],
[[0, 1, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0]],
...
[[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 1]]])
How can i resize a numpy array and fill it with a specific value (if some dimension is extended) ?
I find a way to extend my array with np.pad but I can't shorten it:
>>> import numpy as np
>>> a = np.ndarray((5, 5), dtype=np.uint16)
>>> a
array([[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]], dtype=uint16)
>>> np.pad(a, ((0, 1), (0,3)), mode='constant', constant_values=9)
array([[0, 0, 0, 0, 0, 9, 9, 9],
[0, 0, 0, 0, 0, 9, 9, 9],
[0, 0, 0, 0, 0, 9, 9, 9],
[0, 0, 0, 0, 0, 9, 9, 9],
[0, 0, 0, 0, 0, 9, 9, 9],
[9, 9, 9, 9, 9, 9, 9, 9]], dtype=uint16)
And if i use resize i can't specify the value that I want to use.
>>> a.fill(5)
>>> a.resize((2, 7))
>>> a
array([[5, 5, 5, 5, 5, 5, 5],
[5, 5, 5, 5, 5, 5, 5]], dtype=uint16)
But i would like
>>> a
array([[5, 5, 5, 5, 5, 9, 9],
[5, 5, 5, 5, 5, 9, 9]], dtype=uint16)
After some test I create this function but it's only work when you change x_value or with a lower y_value, if you need to increase y dimension it doesn't work, why ?
VALUE_TO_FILL = 9
def resize(self, x_value, y_value):
x_diff = self.np_array.shape[0] - x_value
y_diff = self.np_array.shape[1] - y_value
self.np_array.resize((x_value, y_value), refcheck=False)
if x_diff < 0:
self.np_array[x_diff:, :] = VALUE_TO_FILL
if y_diff < 0:
self.np_array[:, y_diff:] = VALUE_TO_FILL
Your array has a fixed size data buffer. You can reshape the array without changing that buffer. You can take a slice (view) without changing the buffer. But you can't add values to the array without changing the buffer.
In general resize returns an new array with a new data buffer.
pad is a complex function to handle general cases. But the simplest approach is to create the empty target array, fill it, and then copy the input into the right place.
Alternatively pad could create the fill arrays and concatenate them with the original. But concatenate also makes the empty return and copies.
A do it yourself pad with clipping could be structured as:
n,m = X.shape
R = np.empty((k,l))
R.fill(value)
<calc slices from n,m,k,l>
R[slice1] = X[slice2]
Calculating the slices may require if-else tests or equivalent min/max. You can probably work out those details.
This may be all that is needed
R[:X.shape[0],:X.shape[1]]=X[:R.shape[0],:R.shape[1]]
That's because there's no problem if a slice is larger than the dimension.
In [37]: np.arange(5)[:10]
Out[37]: array([0, 1, 2, 3, 4])
Thus, for example:
In [38]: X=np.ones((3,4),int)
In [39]: R=np.empty((2,5),int)
In [40]: R.fill(9)
In [41]: R[:X.shape[0],:X.shape[1]]=X[:R.shape[0],:R.shape[1]]
In [42]: R
Out[42]:
array([[1, 1, 1, 1, 9],
[1, 1, 1, 1, 9]])
To shorten it, you can use negative values in slice :
>>> import numpy as np
>>> a = np.ndarray((5, 5), dtype=np.uint16)
>>> a
array([[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]], dtype=uint16)
>>> b = a[0:-1,0:-3]
>>> b
array([[0, 0],
[0, 0],
[0, 0],
[0, 0]], dtype=uint16)
There have been a couple questions on SO about how to initialize a 2-dimensional matrix, with the answer being something like this:
matrix = [[0 for x in range(10)] for x in range(10)]
Is there any way to generalize this to n dimensions without using for blocks or writing out a really long nested list comprehension?
As integers are immutable you can reduce your code to:
matrix = [[0] * 10 for x in range(10)]
As #iCodez mentioned in comments if NumPy is an option you can simply do:
import numpy as np
matrix = np.zeros((10, 10))
If you really want a matrix, np.zeros and np.ones can quickly create such a 2 dimensional array for instantiating a matrix:
import numpy as np
my_matrix = np.matrix(np.zeros((10,10)))
To generalize to n dimensions, you can't use a matrix, which by definition is 2 dimensional:
n_dimensions = 3
width = 10
n_dimensional_array = np.ones((width,) * n_dimensions)
#brian-putman was faster and better... anyway, this is my solution:
init = lambda x, y: [init(x, y-1) if y>1 else 0 for _ in xrange(x)]
that generates only square matrices of size x filled with zeroes in y dimensions. called like this
init(5, 3)
[[[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]],
[[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]],
[[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]],
[[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]],
[[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]]]
I agree that if numpy is an option, it's a much easier way to work with matrices. I highly recommend it.
That being said, this recursive function is a reasonable way to generalize your code to n dimensions. The first parameter is a list or tuple specifying how large each dimension should be (and, indirectly, how many dimensions). The second parameter is the constant value to fill the matrix with (in your example, 0):
def init(sizes, value=0):
if (len(sizes) == 1):
return [value] * sizes[0]
else:
# old code - fixed per comment. This method does not create
# sizes[0] *new* lists, it just repeats the same list
# sizes[0] times. This causes unexpected behavior when you
# try to set an item in a list and all of its siblings get
# the same change
# return [init(sizes[1:], value)] * sizes[0]
# this method works better; it creates a new list each time through
return [init(sizes[1:], value) for i in xrange(sizes[0])]
matrix = init((2,3,4), 5)
matrix[0][0][0] = 100 # setting value per request in comment
print matrix
>>> [[[100, 5, 5, 5], [5, 5, 5, 5], [5, 5, 5, 5]], [[5, 5, 5, 5], [5, 5, 5, 5], [5, 5, 5, 5]]]
N-dimensional arrays are a little hard to print on a 2D screen, but you can see the structure of matrix a little more easily in the snippet below which I manually indented. It's an array of length 2, containing arrays of length 3, containing arrays of length 4, where every value is set to 5:
[
[
[100, 5, 5, 5],
[5, 5, 5, 5],
[5, 5, 5, 5]
],
[
[5, 5, 5, 5],
[5, 5, 5, 5],
[5, 5, 5, 5]
]
]