Develop Reference

Combining numpys fancy indexing with slicing

I currently have a two-dimensional numpy array of shape (m, n). Furthermore, I have two (m, p) arrays of indices i1 and i2. The indices are always contiguous!
import numpy as np
t = np.array([[-1, -1, 0, 0, 1, 2, 2],
[-1, -1, 0, 1, 2, 3, 3],
[0, 0, 1, 2, 2, 3, 3]])
i1 = np.array([3, 2, 2])
i2 = np.array([4, 3, 3])
How do I use the arrays i1 and i2 to slice t in order to obtain the following sub-matrix?
expected_t = np.array([
[0, 1],
[0, 1],
[1, 2]
])
That is
expected_t[0, :] = t[0, i1[0]:i2[0]]
expected_t[1, :] = t[1, i1[1]:i2[1]]
expected_t[2, :] = t[2, i1[2]:i2[2]]
Furthermore, is this possible to do without copying the data by creating a view?
Thanks in advance for all help!

Use fancy indexing in numpy:
t[np.arange(3).reshape(3,1), np.vstack((i1,i2)).T]
OR
t[np.arange(3), np.vstack((i1,i2))].T
Both will have the result:
array([[0, 1],
[0, 1],
[1, 2]])

I suggest this, but I don't know if it exists a fastest way of indexing following the example :
import numpy as np
t = np.array([[-1, -1, 0, 0, 1, 2, 2],
[-1, -1, 0, 1, 2, 3, 3],
[0, 0, 1, 2, 2, 3, 3]])
i1 = np.array([3, 2, 2])
i2 = np.array([4, 3, 3])
output = []
for i, (min_, max_) in enumerate(zip(i1, i2)):
output.append(t[i, min_:max_+1])
expected_t = np.array(output)
Or shorter :
expected_t = np.array([t[i, j:k+1] for (i,j,k) in zip(range(len(t)), i1, i2)])

Generating all combinations of 1 and -1 in n Dimensions with numpy?

I need a way to generate a numpy array with all possible combination of [-1, 1] given a number of dimensions.
For example If i have 2 dimensions I would get :
[[1, 1], [1, -1], [-1, 1], [-1, -1]]
If I have 3 dimensions I would get:
[[1, 1, 1], [1, 1, -1], [1, -1, 1], [1, -1, -1], [-1, 1, 1], [-1, 1, -1], [-1, -1, 1], [-1, -1, -1]],
I have tried something like this :
import numpy as np
def permgrid(n):
inds = np.indices((2,) * n)
return inds.reshape(n, -1).T
But this only returns all combinations of 0 and 1.

You can use the product function from itertools.
Basically, you get all the combinations with repeat of 2.
print (list(itertools.product([1,-1], repeat=2)))
itertools.product(*iterables[, repeat])
Cartesian product of input iterables.
Roughly equivalent to nested for-loops in a generator expression.
You can read more in here

Here's a NumPy's broadcasting based method -
def broadcasting_typecast(n):
return -2*((np.arange(2**n)[:,None] & (1 << np.arange(n-1,-1,-1))) != 0)+1
Sample runs -
In [231]: n = 2
In [232]: broadcasting_typecast(n)
Out[232]:
array([[ 1, 1],
[ 1, -1],
[-1, 1],
[-1, -1]])
In [233]: n = 3
In [234]: broadcasting_typecast(n)
Out[234]:
array([[ 1, 1, 1],
[ 1, 1, -1],
[ 1, -1, 1],
[ 1, -1, -1],
[-1, 1, 1],
[-1, 1, -1],
[-1, -1, 1],
[-1, -1, -1]])

Either replace,
def permgrid(n):
inds = np.indices((2,) * n)
out = inds.reshape(n, -1).T
return np.where(out==0, -np.ones_like(out), out)
or do it with math:
def permgrid(n):
inds = np.indices((2,) * n)
return inds.reshape(n, -1).T*2-1

You might want to have a look at itertools. It's a package for the generation of sorted sequences and the like.
import itertools as it
for element in it.combinations_with_replacement([1,-1],3):
print element

you could use np.ix_. advantage: you can easily replace -1,1 with whatever you like (other numbers, other dtypes, more than 2, etc.)
>>> n = 3
>>> out = np.empty(n*(2,)+(n,), dtype=int)
>>> for j, sl in enumerate(np.ix_(*(n*((-1,1),)))):
... out[..., j] = sl
...
>>> out
array([[[[-1, -1, -1],
[-1, -1, 1]],
[[-1, 1, -1],
[-1, 1, 1]]],
[[[ 1, -1, -1],
[ 1, -1, 1]],
[[ 1, 1, -1],
[ 1, 1, 1]]]])
Optionally:
flat_out = np.reshape(out, (-1, n))

Numpy efficient indexing with varied size arrays

Take a look at this piece of code:
import numpy as np
a = np.random.random(10)
indicies = [
np.array([1, 4, 3]),
np.array([2, 5, 8, 7, 3]),
np.array([1, 2]),
np.array([3, 2, 1])
]
result = np.zeros(2)
result[0] = a[indicies[0]].sum()
result[1] = a[indicies[2]].sum()
Is there any way to get result more efficiently? In my case a is a very large array.
In other words I want to select elements from a with several varying size index arrays and then sum over them in one operation, resulting in a single array.

With your a and indicies list:
In [280]: [a[i].sum() for i in indicies]
Out[280]:
[1.3986792680307709,
2.6354365193743732,
0.83324677494990895,
1.8195179021311731]
Which of course could wrapped in np.array().
For a subset of the indicies items use:
In [281]: [a[indicies[i]].sum() for i in [0,2]]
Out[281]: [1.3986792680307709, 0.83324677494990895]
A comment suggests indicies comes from an Adjacency matrix, possibly sparse.
I could recreate such an array with:
In [289]: A=np.zeros((4,10),int)
In [290]: for i in range(4): A[i,indicies[i]]=1
In [291]: A
Out[291]:
array([[0, 1, 0, 1, 1, 0, 0, 0, 0, 0],
[0, 0, 1, 1, 0, 1, 0, 1, 1, 0],
[0, 1, 1, 0, 0, 0, 0, 0, 0, 0],
[0, 1, 1, 1, 0, 0, 0, 0, 0, 0]])
and use a matrix product (np.dot) to do the selection and sum:
In [292]: A.dot(a)
Out[292]: array([ 1.39867927, 2.63543652, 0.83324677, 1.8195179 ])
A[[0,2],:].dot(a) would use a subset of rows.
A sparse matrix version has that list of row indices:
In [294]: Al=sparse.lil_matrix(A)
In [295]: Al.rows
Out[295]: array([[1, 3, 4], [2, 3, 5, 7, 8], [1, 2], [1, 2, 3]], dtype=object)
And a matrix product with that gives the same numbers:
In [296]: Al*a
Out[296]: array([ 1.39867927, 2.63543652, 0.83324677, 1.8195179 ])

If your array a is very large you might have memory issues if your array of indices contains many arrays of many indices when looping through it.
To avoid this issue use an iterator instead of a list :
indices = iter(indices)
and then loop through your iterator.

Matrix multiplying arrays with Numpy

I have a 5x5 array of arrays and I'm trying to matrix multiply the transpose of one row with another row.
import numpy as np
a = np.array([1, 4, 6, 4, 1])
b = np.array([-1, -2, 0, 2, 1])
c = np.array([-1, 2, 0, -2, 1])
d = np.array([-1, 0, 2, 0, -1])
e = np.array([1, -4, 6, -4, 1])
f = np.vstack([a, b, c, d, e])
result = np.dot(f[1, :].T, f[1, :])
I assumed this would work but apparently
f[1, :].T
ends up becoming
[-1, -2, 0, 2, 1]
rather than
[[-1]
[-2]
[ 0]
[ 2]
[ 1]]
and so np.dot treats it like a real dot product rather than doing matrix multiplication.
I found out that list slicing where one index is an integer and all others are :s reduces the dimension by one so so the shape of f[1, :] is not (1, 5) but (5,) and so transposing it does nothing.
I've been able to get it to working using f[1, :].reshape((1, 5)) but is there a better way of doing this? Am I missing a simple way of getting the transpose without having to reshape it?

You can use np.newaxis to add a dimension when slicing, to compensate for the dimension that is otherwise lost.
f[1, :, np.newaxis]
produces the single-column 2D array you want. Putting np.newaxis before the colon would give a single-row 2D array.

For numpy arrays it is often favorable to have this behavior, to circumvent this you can always use the numpy matrix class.
>>> f = np.matrix(f)
>>> f
matrix([[ 1, 4, 6, 4, 1],
[-1, -2, 0, 2, 1],
[-1, 2, 0, -2, 1],
[-1, 0, 2, 0, -1],
[ 1, -4, 6, -4, 1]])
>>> f[1,:].T
matrix([[-1],
[-2],
[ 0],
[ 2],
[ 1]])
>>> np.dot(f[1, :].T, f[1, :])
matrix([[ 1, 2, 0, -2, -1],
[ 2, 4, 0, -4, -2],
[ 0, 0, 0, 0, 0],
[-2, -4, 0, 4, 2],
[-1, -2, 0, 2, 1]])
As this is the matrix class * will denote matrix multiplication, therefore you can simply use:
f[1,:].T * f[1,:]
Also you may want to consider np.outer for this kind of operation:
>>> np.outer(f[1,:],f[1,:])
array([[ 1, 2, 0, -2, -1],
[ 2, 4, 0, -4, -2],
[ 0, 0, 0, 0, 0],
[-2, -4, 0, 4, 2],
[-1, -2, 0, 2, 1]])

If you want the individual slices to retain their "matrixness" then you should cast f to a numpy.matrix, which preserves the matrixness.
fm = numpy.matrix(f)
then
numpy.dot(fm[1,:].T,fm[1,:])
will return an nxn matrix

Following the accepted answer, I prefer to use None instead of np.newaxis, which is a little verbose for my tastes. For example,
f[:,None]
does the same thing as f[:,np.newaxis].

I am late to the party here, it is worth noting that the NumPy webpage I accessed today warns that
class numpy.matrix(data, dtype=None, copy=True)
might be removed.

Binary numpy array to list of integers?

I have a binary array, and I would like to convert it into a list of integers, where each int is a row of the array.
For example:
from numpy import *
a = array([[1, 1, 0, 0], [0, 1, 0, 0], [0, 1, 1, 1], [1, 1, 1, 1]])
I would like to convert a to [12, 4, 7, 15].

#SteveTjoa's answer is fine, but for kicks, here's a numpy one-liner:
In [19]: a
Out[19]:
array([[1, 1, 0, 0],
[0, 1, 0, 0],
[0, 1, 1, 1],
[1, 1, 1, 1]])
In [20]: a.dot(1 << arange(a.shape[-1] - 1, -1, -1))
Out[20]: array([12, 4, 7, 15])
(arange is numpy.arange.)
If the bits are in the opposite order, change the order of the values produced by arange:
In [25]: a.dot(1 << arange(a.shape[-1]))
Out[25]: array([ 3, 2, 14, 15])

I once asked a similar question here. Here was my answer, adapted for your question:
def bool2int(x):
y = 0
for i,j in enumerate(x):
y += j<<i
return y
In [20]: a
Out[20]:
array([[1, 1, 0, 0],
[0, 1, 0, 0],
[0, 1, 1, 1],
[1, 1, 1, 1]])
In [21]: [bool2int(x[::-1]) for x in a]
Out[21]: [12, 4, 7, 15]

You could also do this within numpy directly:
from numpy import *
a = array([[1, 1, 0, 0], [0, 1, 0, 0], [0, 1, 1, 1], [1, 1, 1, 1]])
b2i = 2**arange(a.shape[0]-1, -1, -1)
result = (a*b2i).sum(axis=1) #[12 4 7 15]

If you like working directly with bitwise math, this one should work pretty well.
def bits2int(a, axis=-1):
return np.right_shift(np.packbits(a, axis=axis), 8 - a.shape[axis]).squeeze()
bits2int(a)
Out: array([12, 4, 7, 15], dtype=uint8)

Another one:
def row_bits2int(arr):
n = arr.shape[1] # number of columns
# shift the bits of the first column to the left by n - 1
a = arr[:, 0] << n - 1
for j in range(1, n):
# "overlay" with the shifted bits of the next column
a |= arr[:, j] << n - 1 - j
return a