Related
import numpy as np
import itertools as it
SPIN_POS = np.array([[0, 0, 0], [1, 1, 0], [1, 0, 1], [0, 1, 1],
[2, 2, 0], [3, 3, 0], [3, 2, 1], [2, 3, 1],
[2, 0, 2], [3, 1, 2], [3, 0, 3], [2, 1, 3],
[0, 2, 2], [1, 3, 2], [1, 2, 3], [0, 3, 3]
]) / 4
def gen_posvecs(xdim:int, ydim:int, zdim:int):
"""
Generates position vectors of site pairs in the lattice of size xdim,ydim,zdim
:param x,y,z is the number of unit cells in the x,y,z directions;
:returns array containing the position vectors
"""
poss = np.zeros((xdim,ydim,zdim,16,3))
for x,y,z,s in it.product(range(xdim), range(ydim), range(zdim), range(16)):
poss[x,y,z,s] = np.array([x,y,z]) + SPIN_POS[s]
return poss
A = gen_sepvecs(4,4,4) # A.shape = (4,4,4,16,3)
B = np.subtract.outer(A[...,-1], A) # my attempt at a soln
assert all(A[1,2,0,12] - A[0,1,3,11] == B[1,2,0,12,0,1,3,11]) # should give true
Consider the above code. I have an array A of shape (4,4,4,16,3), which represents 3D position vectors in a lattice (the last axis of dim 3 are the x,y,z coordinates). The first 4 dimensions index the site in the lattice.
What I want
I would like to generate from A, an array containing all possible separation vectors between sites in the lattice. This means an output array B, of shape (4,4,4,16,4,4,4,16,3). The first 4 dimensions being of site i, next 4 dimensions of site j, then the last dimension of the (x,y,z) coordinate of the position vector difference.
i.e., A[a,b,c,d]: shape (3,) is the (x,y,z) of first site; A[r,s,t,u]: shape (3,) is the (x,y,z) of second site; Then I want B[a,b,c,d,r,s,t,u] to be (x,y,z) difference between the first two.
My attempt
I know about the ufunc.outer function, as you can see in my attempt in code. But I'm stuck at applying it together with performing element-wise subtraction on the last axis (the (x,y,z)) of each A.
In my attempt, B has the correct dimensions I want, but it is obviously wrong. Any hints? (barring the use of any for-loops)
I think you just need to do:
B = (A[:, :, :, :, np.newaxis, np.newaxis, np.newaxis, np.newaxis] -
A[np.newaxis, np.newaxis, np.newaxis, np.newaxis])
In your code:
import numpy as np
import itertools as it
SPIN_POS = np.array([[0, 0, 0], [1, 1, 0], [1, 0, 1], [0, 1, 1],
[2, 2, 0], [3, 3, 0], [3, 2, 1], [2, 3, 1],
[2, 0, 2], [3, 1, 2], [3, 0, 3], [2, 1, 3],
[0, 2, 2], [1, 3, 2], [1, 2, 3], [0, 3, 3]
]) / 4
def gen_posvecs(xdim:int, ydim:int, zdim:int):
"""
Generates position vectors of site pairs in the lattice of size xdim,ydim,zdim
:param x,y,z is the number of unit cells in the x,y,z directions;
:returns array containing the position vectors
"""
poss = np.zeros((xdim,ydim,zdim,16,3))
for x,y,z,s in it.product(range(xdim), range(ydim), range(zdim), range(16)):
poss[x,y,z,s] = np.array([x,y,z]) + SPIN_POS[s]
return poss
A = gen_posvecs(4,4,4) # A.shape = (4,4,4,16,3)
B = A[:, :, :, :, np.newaxis, np.newaxis, np.newaxis, np.newaxis] - A[np.newaxis, np.newaxis, np.newaxis, np.newaxis]
assert all(A[1,2,0,12] - A[0,1,3,11] == B[1,2,0,12,0,1,3,11])
# Does not fail
import numpy as np
a = np.array([[1,2], [3, 4], [5, 6]])
print(a[[0, 1, 2], [0, 1, 0]]) # Prints "[1 4 5]"
print(a[[0, 0], [1, 1]]) # Prints "[2 2]"
I don't understand why it results [1 4 5] and [2 2]
Because you're slicing the array with indexes
a[[0, 1, 2], [0, 1, 0]] is equivalent to
a[0, 0] # 1
a[1, 1] # 4
a[2, 0] # 5
whereas a[[0, 0], [1, 1]] is equivalent to twice a[0, 1]
More about Numpy indexing here
Think of it as 2d-array access. When you initialize a you get your 2d array in the form:
[ 1 2 ]
[ 3 4 ]
[ 5 6 ]
Numpy indexing when given a 2d array works as follows: you input a list of the row indexes, then a list of the column indexes. Semantically your first index retrieval statement is saying "from row 0 retrieve element 0, from row 1 retrieve element 1, and from row 2 retrieve element 0" which corresponds to [1 4 5]. You can then figure out why you get [2 2] for the second statement.
You can read more about this advanced indexing here: https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#integer-array-indexing
From Getting indices of both zero and nonzero elements in array, I can get indicies of non-zero elements in a 1 D array in numpy like this:
indices_nonzero = numpy.arange(len(array))[~bindices_zero]
Is there a way to extend it to a 2D array?
You can use numpy.nonzero
The following code is self-explanatory
import numpy as np
A = np.array([[1, 0, 1],
[0, 5, 1],
[3, 0, 0]])
nonzero = np.nonzero(A)
# Returns a tuple of (nonzero_row_index, nonzero_col_index)
# That is (array([0, 0, 1, 1, 2]), array([0, 2, 1, 2, 0]))
nonzero_row = nonzero[0]
nonzero_col = nonzero[1]
for row, col in zip(nonzero_row, nonzero_col):
print("A[{}, {}] = {}".format(row, col, A[row, col]))
"""
A[0, 0] = 1
A[0, 2] = 1
A[1, 1] = 5
A[1, 2] = 1
A[2, 0] = 3
"""
You can even do this
A[nonzero] = -100
print(A)
"""
[[-100 0 -100]
[ 0 -100 -100]
[-100 0 0]]
"""
Other variations
np.where(array)
It is equivalent to np.nonzero(array)
But, np.nonzero is preferred because its name is clear
np.argwhere(array)
It's equivalent to np.transpose(np.nonzero(array))
print(np.argwhere(A))
"""
[[0 0]
[0 2]
[1 1]
[1 2]
[2 0]]
"""
A = np.array([[1, 0, 1],
[0, 5, 1],
[3, 0, 0]])
np.stack(np.nonzero(A), axis=-1)
array([[0, 0],
[0, 2],
[1, 1],
[1, 2],
[2, 0]])
np.nonzero returns a tuple of arrays, one for each dimension of a, containing the indices of the non-zero elements in that dimension.
https://docs.scipy.org/doc/numpy/reference/generated/numpy.nonzero.html
np.stack joins this tuple array along a new axis. In our case, the innermost axis also known as the last axis (denoted by -1).
The axis parameter specifies the index of the new axis in the dimensions of the result. For example, if axis=0 it will be the first dimension and if axis=-1 it will be the last dimension.
New in version 1.10.0.
https://docs.scipy.org/doc/numpy/reference/generated/numpy.stack.html
I'm moving my first steps with theano and I cannot figure out how to solve this problem which could be actually very easy.
I have a 3 * 4 * 2 tensor, like the following:
[1 1] | [2 2] | [3 3]
[1 1] | [2 2] | [3 3]
[0 0] | [2 2] | [3 3]
[9 9] | [0 0] | [3 3]
So I have N=3 sequences, each of them of length L=4 with their elements that are vectors of dimension d=2. Actually, the sequences can be of different length but I could think of padding them with [0 0] vectors, as shown above.
What I want to do is, first scan through the first axis of the tensor and sum up all the vector in the lists up to the the first [0 0] vector -- that's why I added the [9 9] at the end of the first tensor slice, in order to check the sum exit condition [1]. I should end up in [[2 2], [6 6], [12 12]]. I tried in many ways to solve this problem which seems to me just a nested looping problem... but always got some weird errors[2].
Thanks,
Giulio
--
[1]: the actual problem is the training of a recurrent neural network for NLP purposes, with N the dimension of the batch, L the max length of a sentence in the batch and d the dimension of the representation of each word. I omitted the problem so that I could focus on the simplest coding aspect.
[2] I omit the history of my failures, maybe I could add them later.
If your sequences are always zero padded then you can just sum along the axis of interest since the padding regions will not change the sum. However, if the padding regions may contain non-zero values there are two approaches.
Use scan. This is slow and should be avoided if possible. In fact it can be avoided because,
Create a binary mask and multiply out the padding region.
Here's some code that illustrates these three approaches. For the two approaches that allow for non-zero padding regions (v2 and v3) the computation needs an additional input: a vector giving the lengths of the sequences within the batch.
import numpy
import theano
import theano.tensor as tt
def v1():
# NOTE: [9, 9] element changed to [0, 0]
# since zero padding must be used for
# this method
x_data = [[[1, 1], [1, 1], [0, 0], [0, 0]],
[[2, 2], [2, 2], [2, 2], [0, 0]],
[[3, 3], [3, 3], [3, 3], [3, 3]]]
x = tt.tensor3()
x.tag.test_value = x_data
y = x.sum(axis=1)
f = theano.function([x], outputs=y)
print f(x_data)
def v2_step(i_t, s_tm1, x, l):
in_sequence = tt.lt(i_t, l).dimshuffle(0, 'x')
s_t = s_tm1 + tt.switch(in_sequence, x[i_t], 0)
return s_t
def v2():
x_data = [[[1, 1], [1, 1], [0, 0], [9, 9]],
[[2, 2], [2, 2], [2, 2], [0, 0]],
[[3, 3], [3, 3], [3, 3], [3, 3]]]
l_data = [2, 3, 4]
x = tt.tensor3()
x.tag.test_value = x_data
l = tt.lvector()
l.tag.test_value = l_data
# Must dimshuffle first because scan can only iterate over first (0'th) axis.
x_hat = x.dimshuffle(1, 0, 2)
y, _ = theano.scan(v2_step, sequences=[tt.arange(x_hat.shape[0])],
outputs_info=[tt.zeros_like(x_hat[0])],
non_sequences=[x_hat, l], strict=True)
f = theano.function([x, l], outputs=y[-1])
print f(x_data, l_data)
def v3():
x_data = [[[1, 1], [1, 1], [0, 0], [9, 9]],
[[2, 2], [2, 2], [2, 2], [0, 0]],
[[3, 3], [3, 3], [3, 3], [3, 3]]]
l_data = [2, 3, 4]
x = tt.tensor3()
x.tag.test_value = x_data
l = tt.lvector()
l.tag.test_value = l_data
indexes = tt.arange(x.shape[1]).dimshuffle('x', 0)
mask = tt.lt(indexes, l.dimshuffle(0, 'x')).dimshuffle(0, 1, 'x')
y = (mask * x).sum(axis=1)
f = theano.function([x, l], outputs=y)
print f(x_data, l_data)
def main():
theano.config.compute_test_value = 'raise'
v1()
v2()
v3()
main()
In general, if your step function is dependent on the output of a previous step then you need to use scan.
If every step/iteration could, in principle, be executed concurrently (i.e. they don't rely on each other at all) then there is often a much more efficient way to do this without using scan
As input, I receive two types of lists of lists made of x and y coordinates that represent polygon and multipolygon geometries. In fact the input is represented in the GeoJson standard
list1 represents coordinates of a simple polygon geometry and list2 represent a multipolygon geometry:
list1 = [[[0 , 0], [0, 1], [0 ,2]]]
list2 = [[[[0, 0] , [0, 1], [0, 2]], [[1, 0], [1, 1], [1 ,2]]]]
Multipolygon geometry (list2) are represented by a list of lists one level deeper than simple polygon geometry (list1).
I want to flatten those lists in order to get those output:
if input is list1 type : list1_out = [[0, 0, 0, 1, 0, 2]]
if input is list2 type : list2_out = [[0, 0, 0, 1, 0, 2], [1, 0, 1, 1, 1, 2]]
I am using the following code that is usually used to flatten lists where input can be a list of the two types:
[coords for polygon in input for coords in polygon]
With this code above, the output for list1 is correct but the output of list2 is the following:
[[[0, 0] ,[0, 1], [0, 2]], [1, 0], [1, 1], [1, 2]]]
Is there a function that could deeply flatten those two types of lists to get the expected output?
Edit: Performance really matter here as the lists are really big
Edit 2: I can use a if sentence to filter each type of list
Try;
for list1
[sum(x, []) for x in list1]
for list2
[sum(x, []) for a in list2 for x in a]
Demo
>>> list1 = [[[0 , 0], [0, 1], [0 ,2]]]
>>> list2 = [[[[0, 0] , [0, 1], [0, 2]], [[1, 0], [1, 1], [1 ,2]]]]
>>> [sum(x, []) for x in list1]
[[0, 0, 0, 1, 0, 2]]
>>> [sum(x, []) for a in list2 for x in a]
[[0, 0, 0, 1, 0, 2], [1, 0, 1, 1, 1, 2]]
>>>
Casting your data to numpy.array, you can use reshape:
import numpy as np
t = np.array([[[[0, 0] , [0, 1], [0, 2]], [[1, 0], [1, 1], [1 ,2]]]])
print t.shape # (1, 2, 3, 2)
t = np.reshape([1, 2, 6]) # merging the last 2 coordinates/axes
flattens the second list as you want.
A code which works for both list (since in both cases you want to merge the last to axis) is:
t = np.array(yourList)
newShape = t.shape[:-2] + (t.shape[-2] * t.shape[-1], ) # this assumes your
# arrays are always at least 2 dimensional (no need to flatten them otherwise...)
t = t.reshape(t, newShape)
The key thing is to keep the shape unchanged up to the last 2 axes (hence
t.shape[:-2]), but to merge the two last axes together (using an axis of length t.shape[-2] * t.shape[-1])
We are creating the new shape by concatenation of these two tuples (hence the extra comma after the multiplication).
Edit: np.reshape() doc is here. The important parameters are the input array (your list, cast as an array), and a tuple which I've called newShape, which represents the lengths along the new axes.