Develop Reference

How would you unionize N-arrays with different sizes?

The :
np.union1d(a, b)
can unionize two arrays with different sizes.
np.vstack((a, b, c)).T.ravel()
can unionize N arrays of the same size.
How would you unionize N-arrays with different sizes ?
And of course it should be fast ;) !
btw union is not just concatenation...
still testing, but would this do it :
np.unique(np.concatenate((a,b,c)))

Here's one with array-assignment + masking for positive numbers -
def unionize_ndarrays(L, maxnum=None):
if maxnum is None:
maxnum = max([np.max(i) for i in L])+1
# for lists : max([max(i) for i in L])+1
id_ar = np.zeros(maxnum, dtype=bool)
for i in L:
id_ar[i] = True
return np.flatnonzero(id_ar)
Computing the max number maxnum has noticeable runtime and could be the bottleneck even for a large number of small arrays. So, if that's known, feeding that in should help a lot on those scenarios.
Sample run -
In [43]: a = np.array([0, 1, 3, 4, 3])
...: b = np.array([0, 10, 3, 1, 2, 1])
...: c = np.array([6, 3, 4, 2])
In [44]: np.unique(np.concatenate((a,b,c)))
Out[44]: array([ 0, 1, 2, 3, 4, 6, 10])
In [45]: unionize_ndarrays((a,b,c))
Out[45]: array([ 0, 1, 2, 3, 4, 6, 10])
Benchmarking
1) Small sized arrays -
In [106]: L = [np.random.randint(0,10,n) for n in np.random.randint(4,10,10000)]
In [107]: %timeit unionize_ndarrays(L, maxnum=10)
2.74 ms ± 207 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [108]: %timeit np.unique(np.concatenate((L)))
3.06 ms ± 24.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# Without maxnum fed
In [109]: %timeit unionize_ndarrays(L)
40.4 ms ± 542 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
If order is not important, we can also look into pandas.factorize, if we are dealing with small-sized arrays -
In [76]: a = np.array([0, 1, 3, 4, 3])
...: b = np.array([0, 10, 3, 1, 2, 1])
...: c = np.array([6, 3, 4, 2])
In [77]: L = [a,b,c]
In [80]: import pandas as pd
In [81]: pd.factorize(np.concatenate(L))[1]
Out[81]: array([ 0, 1, 3, 4, 10, 2, 6])
Related timings -
In [82]: L = [np.random.randint(0,10,n) for n in np.random.randint(4,10,10000)]
In [84]: %timeit pd.factorize(np.concatenate(L))[1]
2.1 ms ± 13.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
2) Big-sized (bigger variation in sizes) arrays -
Timings -
In [2]: L = [np.random.randint(0,1000,n) for n in np.random.randint(10,1000,10000)]
In [3]: %timeit unionize_ndarrays(L, maxnum=1000)
...: %timeit unionize_ndarrays(L)
...: %timeit np.unique(np.concatenate((L)))
14 ms ± 925 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
56.6 ms ± 641 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
242 ms ± 773 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
So, to choose one will depend on whether we have the priori info on max number and the size variation.

From NumPy manual:
To find the union of more than two arrays, use functools.reduce:
>>> from functools import reduce
>>> reduce(np.union1d, ([1, 3, 4, 3], [3, 1, 2, 1], [6, 3, 4, 2]))
array([1, 2, 3, 4, 6])
This also works with arrays of different sizes:
>>> reduce(np.union1d, ([0, 1, 3, 4, 3], [0, 10, 3, 1, 2, 1], [6, 3, 4, 2]))
array([ 0, 1, 2, 3, 4, 6, 10])

Efficiently apply different permutations for each row of a 2D NumPy array [duplicate]

This question already has answers here:
Randomly shuffle items in each row of numpy array
(6 answers)
Closed 3 years ago.
Given a matrix A, I want to apply different random shuffles for different row of A; for example,
array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
becomes
array([[1, 3, 2],
[6, 5, 4],
[7, 9, 8]])
Of course we can loop through the matrix and make every row randomly shuffle; however iteration is slow and I am asking if there is more efficient way to do this.

Picked up this neat trick from Divakar which involves randn and argsort:
np.random.seed(0)
s = np.arange(16).reshape(4, 4)
np.take_along_axis(s, np.random.randn(*s.shape).argsort(axis=1), axis=1)
array([[ 1, 0, 3, 2],
[ 4, 6, 5, 7],
[11, 10, 8, 9],
[14, 12, 13, 15]])
For a 2D array, this can be simplified to
s[np.arange(len(s))[:,None], np.random.randn(*s.shape).argsort(axis=1)]
array([[ 1, 0, 3, 2],
[ 4, 6, 5, 7],
[11, 10, 8, 9],
[14, 12, 13, 15]])
You can also apply np.random.permutation over each row independently to return a new array.
np.apply_along_axis(np.random.permutation, axis=1, arr=s)
array([[ 3, 1, 0, 2],
[ 4, 6, 5, 7],
[ 8, 9, 10, 11],
[15, 14, 13, 12]])
Performance -
s = np.arange(10000 * 100).reshape(10000, 100)
%timeit s[np.arange(len(s))[:,None], np.random.randn(*s.shape).argsort(axis=1)]
%timeit np.apply_along_axis(np.random.permutation, 1, s)
84.6 ms ± 857 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
842 ms ± 8.06 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
I've noticed it depends on the dimensions of your data, make sure to test it out first.

Codewise you can use numpy's apply_along_axis as
np.apply_along_axis(np.random.shuffle, 1, matrix)
but it doesn't seem to be more efficient than iterating at least for a 3x3 matrix, for that method I get
> %%timeit
> np.apply_along_axis(np.random.shuffle, 1, test)
67 µs ± 1.8 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
while the iteration gives
> %%timeit
> for i in range(test.shape[0]):
> np.random.shuffle(test[i])
20.3 µs ± 284 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Numpy: create a matrix from a cartesian product of two vectors (or one with itself) while applying a function to all pairs

To give a bit of explanation, I want to create a covariance matrix where each element is defined by a kernel function k(x, y), and I want to do this for a single vector. It should be something like:
# This is given
x = [x1, x2, x3, x4, ...]
# This is what I want to compute
result = [[k(x1, x1), k(x1, x2), k(x1, x3), ...],
[k(x2, x1), k(x2, x2), ...],
[k(x3, x1), k(x3, x2), ...],
...]
but of course this should be done in numpy arrays, ideally without doing Python interations, because of performance. If I didn't care about performance, I'd probably just write:
result = np.zeros((len(x), len(x)))
for i in range(len(x)):
for j in range(len(x)):
result[i, j] = k(x[i], x[j])
But I feel like there must be a more idiomatic way to write this pattern.

If k operates on 2D arrays, you could use np.meshgrid. But, this would have extra memory overhead. One alternative would be to create 2D mesh views same as with np.meshgrid, like so -
def meshgrid1D_view(x):
shp = (len(x),len(x))
mesh1 = np.broadcast_to(x,shp)
mesh2 = np.broadcast_to(x[:,None],shp)
return mesh1, mesh2
Sample run -
In [140]: x
Out[140]: array([3, 5, 6, 8])
In [141]: np.meshgrid(x,x)
Out[141]:
[array([[3, 5, 6, 8],
[3, 5, 6, 8],
[3, 5, 6, 8],
[3, 5, 6, 8]]), array([[3, 3, 3, 3],
[5, 5, 5, 5],
[6, 6, 6, 6],
[8, 8, 8, 8]])]
In [142]: meshgrid1D(x)
Out[142]:
(array([[3, 5, 6, 8],
[3, 5, 6, 8],
[3, 5, 6, 8],
[3, 5, 6, 8]]), array([[3, 3, 3, 3],
[5, 5, 5, 5],
[6, 6, 6, 6],
[8, 8, 8, 8]]))
How does this help?
It helps with memory efficiency and hence performance. Let's test out on large arrays to see the difference -
In [143]: x = np.random.randint(0,10,(10000))
In [144]: %timeit np.meshgrid(x,x)
10 loops, best of 3: 171 ms per loop
In [145]: %timeit meshgrid1D(x)
100000 loops, best of 3: 6.91 µs per loop

Another solution is to let numpy do the broadcasting itself:
import numpy as np
def k(x,y):
return x**2+y
def meshgrid1D_view(x):
shp = (len(x),len(x))
mesh1 = np.broadcast_to(x,shp)
mesh2 = np.broadcast_to(x[:,None],shp)
return mesh1, mesh2
x = np.random.randint(0,10,(10000))
b=k(a[:,None],a[None,:])
def sol0(x):
k(x[:,None],x[None,:])
def sol1(x):
x,y=np.meshgrid(x,x)
k(x,y)
def sol2(x):
x,y=meshgrid1D_view(x)
k(x,y)
%timeit sol0(x)
165 ms ± 1.67 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit sol1(x)
655 ms ± 6.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit sol2(x)
341 ms ± 2.91 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
You see this is more efficient and there's less code.

Python numpy, reshape/transform array avoiding iteration through rows

I have a time series with 4 features at each step, it looks like a set of rows with 4 columns. I want convert it, so row N will contain a vector of features of rows N and N-1
a = np.array([[1,2,3,0], [4,5,6,0], [7,8,9,0], [10,11,12,0]])
array([[ 1, 2, 3, 0],
[ 4, 5, 6, 0],
[ 7, 8, 9, 0],
[10, 11, 12, 0]])
a.shape
(4, 4)
convert to:
array([[[ 1, 2, 3, 0],
[ 4, 5, 6, 0]],
[[ 4, 5, 6, 0],
[ 7, 8, 9, 0]],
[[ 7, 8, 9, 0],
[10, 11, 12, 0]]])
a_.shape
(3, 2, 4)
I'm using the following code to do that:
seq_len = 2
for i in range(seq_len, a.shape[0]+1):
if i-seq_len == 0:
a_ = a[i-seq_len:i, :].reshape(1, -1, 4)
else:
a_ = np.vstack([a_, a[i-seq_len:i, :].reshape(1, -1, 4)])
It's working but I think it is not an optimal solution. Could you please suggest how I can improve my code by avoiding 'for' cycle?

Use adequate slicing and np.stack along the adequate axis.
np.stack((a[:-1], a[1:]), axis=1)
Some timings to compare with the other answer out there.
In [13]: s = 1_000_000
In [15]: a = np.arange(s).reshape((s//4,4))
In [21]: %timeit a[[(i-1,i) for i in range(1,a.shape[0])],:]
127 ms ± 724 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [22]: %timeit np.stack((a[:-1], a[1:]), axis=1) # My solution
6.8 ms ± 8.18 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Avoiding any python-level for-loop is the way to go, OP was right.

Use slicing: a[[(i-1,i) for i in range(1,a.shape[0])],:]
Edit: nicoco's answer is the better one.

how to create an array of specified dimension of specific type initialized with same value in python?

I wanna create some array in python of array of specified dimension of specific type initialized with same value. i can use numpy arrays of specific size but I am not sure how to initialize them with a specific value. Off course I don't want to use zeros() or ones()
Thanks a lot.

There are lots of ways to do this. The first one-liner that occurred to me is tile:
>>> numpy.tile(2, 25)
array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2])
You can tile a value in any shape:
>>> numpy.tile(2, (5, 5))
array([[2, 2, 2, 2, 2],
[2, 2, 2, 2, 2],
[2, 2, 2, 2, 2],
[2, 2, 2, 2, 2],
[2, 2, 2, 2, 2]])
However, as a number of answers below indicate, this isn't the fastest method. It's designed for tiling arrays of any size, not just single values, so if you really just want to fill an array with a single value, then it's much faster to allocate the array first, and then use slice assignment:
>>> a = numpy.empty((5, 5), dtype=int)
>>> a[:] = 2
>>> a
array([[2, 2, 2, 2, 2],
[2, 2, 2, 2, 2],
[2, 2, 2, 2, 2],
[2, 2, 2, 2, 2],
[2, 2, 2, 2, 2]])
According to a few tests I did, there aren't any faster approaches. However, two of the approaches mentioned in answers below are equally fast: ndarray.fill and numpy.full.
These tests were all done in ipython, using Python 3.6.1 on a newish mac running OS 10.12.6. Definitions:
def fill_tile(value, shape):
return numpy.tile(value, shape)
def fill_assign(value, shape, dtype):
new = numpy.empty(shape, dtype=dtype)
new[:] = value
return new
def fill_fill(value, shape, dtype):
new = numpy.empty(shape, dtype=dtype)
new.fill(value)
return new
def fill_full(value, shape, dtype):
return numpy.full(shape, value, dtype=dtype)
def fill_plus(value, shape, dtype):
new = numpy.zeros(shape, dtype=dtype)
new += value
return new
def fill_plus_oneline(value, shape, dtype):
return numpy.zeros(shape, dtype=dtype) + value
for f in [fill_assign, fill_fill, fill_full, fill_plus, fill_plus_oneline]:
assert (fill_tile(2, (500, 500)) == f(2, (500, 500), int)).all()
tile is indeed quite slow:
In [3]: %timeit fill_tile(2, (500, 500))
947 µs ± 10.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
Slice assignment ties with ndarray.fill and numpy.full for first place:
In [4]: %timeit fill_assign(2, (500, 500), int)
102 µs ± 1.37 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [5]: %timeit fill_fill(2, (500, 500), int)
102 µs ± 1.99 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [6]: %timeit fill_full(2, (500, 500), int)
102 µs ± 1.47 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In-place broadcasted addition is only slightly slower:
In [7]: %timeit fill_plus(2, (500, 500), int)
179 µs ± 3.7 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
And non-in-place broadcasted addition is only slightly slower than that:
In [8]: %timeit fill_plus_oneline(2, (500, 500), int)
213 µs ± 4.74 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

How about:
shape = (100,100)
val = 3.14
dt = np.float
a = np.empty(shape,dtype=dt)
a.fill(val)
This way you can set things and pass the parameters in. Also, in terms of timings
In [35]: %timeit a=np.empty(shape,dtype=dt); a.fill(val)
100000 loops, best of 3: 13 us per loop
In [36]: %timeit a=np.tile(val,shape)
10000 loops, best of 3: 102 us per loop
So using empty with fill seems significantly faster than tile.

As of NumPy 1.8, you can use numpy.full() to achieve this.
>>> import numpy as np
>>> np.full((3,4), 100, dtype = int)
array([[ 100, 100, 100, 100],
[ 100, 100, 100, 100],
[ 100, 100, 100, 100]])

Are you looking for something like this?
>>> [3 for x in range(10)]
[3, 3, 3, 3, 3, 3, 3, 3, 3, 3]
You can pass the resulting array to numpy.array.