Sometimes it is useful to "clone" a row or column vector to a matrix. By cloning I mean converting a row vector such as
[1, 2, 3]
Into a matrix
[[1, 2, 3],
[1, 2, 3],
[1, 2, 3]]
or a column vector such as
[[1],
[2],
[3]]
into
[[1, 1, 1]
[2, 2, 2]
[3, 3, 3]]
In MATLAB or octave this is done pretty easily:
x = [1, 2, 3]
a = ones(3, 1) * x
a =
1 2 3
1 2 3
1 2 3
b = (x') * ones(1, 3)
b =
1 1 1
2 2 2
3 3 3
I want to repeat this in numpy, but unsuccessfully
In [14]: x = array([1, 2, 3])
In [14]: ones((3, 1)) * x
Out[14]:
array([[ 1., 2., 3.],
[ 1., 2., 3.],
[ 1., 2., 3.]])
# so far so good
In [16]: x.transpose() * ones((1, 3))
Out[16]: array([[ 1., 2., 3.]])
# DAMN
# I end up with
In [17]: (ones((3, 1)) * x).transpose()
Out[17]:
array([[ 1., 1., 1.],
[ 2., 2., 2.],
[ 3., 3., 3.]])
Why wasn't the first method (In [16]) working? Is there a way to achieve this task in python in a more elegant way?
Use numpy.tile:
>>> tile(array([1,2,3]), (3, 1))
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
or for repeating columns:
>>> tile(array([[1,2,3]]).transpose(), (1, 3))
array([[1, 1, 1],
[2, 2, 2],
[3, 3, 3]])
Here's an elegant, Pythonic way to do it:
>>> array([[1,2,3],]*3)
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
>>> array([[1,2,3],]*3).transpose()
array([[1, 1, 1],
[2, 2, 2],
[3, 3, 3]])
the problem with [16] seems to be that the transpose has no effect for an array. you're probably wanting a matrix instead:
>>> x = array([1,2,3])
>>> x
array([1, 2, 3])
>>> x.transpose()
array([1, 2, 3])
>>> matrix([1,2,3])
matrix([[1, 2, 3]])
>>> matrix([1,2,3]).transpose()
matrix([[1],
[2],
[3]])
First note that with numpy's broadcasting operations it's usually not necessary to duplicate rows and columns. See this and this for descriptions.
But to do this, repeat and newaxis are probably the best way
In [12]: x = array([1,2,3])
In [13]: repeat(x[:,newaxis], 3, 1)
Out[13]:
array([[1, 1, 1],
[2, 2, 2],
[3, 3, 3]])
In [14]: repeat(x[newaxis,:], 3, 0)
Out[14]:
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
This example is for a row vector, but applying this to a column vector is hopefully obvious. repeat seems to spell this well, but you can also do it via multiplication as in your example
In [15]: x = array([[1, 2, 3]]) # note the double brackets
In [16]: (ones((3,1))*x).transpose()
Out[16]:
array([[ 1., 1., 1.],
[ 2., 2., 2.],
[ 3., 3., 3.]])
Let:
>>> n = 1000
>>> x = np.arange(n)
>>> reps = 10000
Zero-cost allocations
A view does not take any additional memory. Thus, these declarations are instantaneous:
# New axis
x[np.newaxis, ...]
# Broadcast to specific shape
np.broadcast_to(x, (reps, n))
Forced allocation
If you want force the contents to reside in memory:
>>> %timeit np.array(np.broadcast_to(x, (reps, n)))
10.2 ms ± 62.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> %timeit np.repeat(x[np.newaxis, :], reps, axis=0)
9.88 ms ± 52.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> %timeit np.tile(x, (reps, 1))
9.97 ms ± 77.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
All three methods are roughly the same speed.
Computation
>>> a = np.arange(reps * n).reshape(reps, n)
>>> x_tiled = np.tile(x, (reps, 1))
>>> %timeit np.broadcast_to(x, (reps, n)) * a
17.1 ms ± 284 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> %timeit x[np.newaxis, :] * a
17.5 ms ± 300 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> %timeit x_tiled * a
17.6 ms ± 240 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
All three methods are roughly the same speed.
Conclusion
If you want to replicate before a computation, consider using one of the "zero-cost allocation" methods. You won't suffer the performance penalty of "forced allocation".
I think using the broadcast in numpy is the best, and faster
I did a compare as following
import numpy as np
b = np.random.randn(1000)
In [105]: %timeit c = np.tile(b[:, newaxis], (1,100))
1000 loops, best of 3: 354 µs per loop
In [106]: %timeit c = np.repeat(b[:, newaxis], 100, axis=1)
1000 loops, best of 3: 347 µs per loop
In [107]: %timeit c = np.array([b,]*100).transpose()
100 loops, best of 3: 5.56 ms per loop
about 15 times faster using broadcast
One clean solution is to use NumPy's outer-product function with a vector of ones:
np.outer(np.ones(n), x)
gives n repeating rows. Switch the argument order to get repeating columns. To get an equal number of rows and columns you might do
np.outer(np.ones_like(x), x)
You can use
np.tile(x,3).reshape((4,3))
tile will generate the reps of the vector
and reshape will give it the shape you want
Returning to the original question
In MATLAB or octave this is done pretty easily:
x = [1, 2, 3]
a = ones(3, 1) * x
...
In numpy it's pretty much the same (and easy to memorize too):
x = [1, 2, 3]
a = np.tile(x, (3, 1))
If you have a pandas dataframe and want to preserve the dtypes, even the categoricals, this is a fast way to do it:
import numpy as np
import pandas as pd
df = pd.DataFrame({1: [1, 2, 3], 2: [4, 5, 6]})
number_repeats = 50
new_df = df.reindex(np.tile(df.index, number_repeats))
Another solution
>> x = np.array([1,2,3])
>> y = x[None, :] * np.ones((3,))[:, None]
>> y
array([[ 1., 2., 3.],
[ 1., 2., 3.],
[ 1., 2., 3.]])
Why? Sure, repeat and tile are the correct way to do this. But None indexing is a powerful tool that has many times let me quickly vectorize an operation (though it can quickly be very memory expensive!).
An example from my own code:
# trajectory is a sequence of xy coordinates [n_points, 2]
# xy_obstacles is a list of obstacles' xy coordinates [n_obstacles, 2]
# to compute dx, dy distance between every obstacle and every pose in the trajectory
deltas = trajectory[:, None, :2] - xy_obstacles[None, :, :2]
# we can easily convert x-y distance to a norm
distances = np.linalg.norm(deltas, axis=-1)
# distances is now [timesteps, obstacles]. Now we can for example find the closest obstacle at every point in the trajectory by doing
closest_obstacles = np.argmin(distances, axis=1)
# we could also find how safe the trajectory is, by finding the smallest distance over the entire trajectory
danger = np.min(distances)
To answer the actual question, now that nearly a dozen approaches to working around a solution have been posted: x.transpose reverses the shape of x. One of the interesting side-effects is that if x.ndim == 1, the transpose does nothing.
This is especially confusing for people coming from MATLAB, where all arrays implicitly have at least two dimensions. The correct way to transpose a 1D numpy array is not x.transpose() or x.T, but rather
x[:, None]
or
x.reshape(-1, 1)
From here, you can multiply by a matrix of ones, or use any of the other suggested approaches, as long as you respect the (subtle) differences between MATLAB and numpy.
import numpy as np
x=np.array([1,2,3])
y=np.multiply(np.ones((len(x),len(x))),x).T
print(y)
yields:
[[ 1. 1. 1.]
[ 2. 2. 2.]
[ 3. 3. 3.]]
Related
I'm a numpy baby and am looking at using numpy.vectorise() to compute a distance matrix. I think that a key part of this is the signature param, but when I run the code below I get an error:
import numpy as np
from scipy.spatial.distance import jaccard
#find jaccard dissimilarities for a constant 1 row * m columns array vs each array in an n rows * m columns nested array, outputting a 1 row * n columns array of dissimilarities
vectorised_compute_jac = np.vectorize(jaccard, signature = '(m),(n,m)->(n)')
array_list = [[1, 2, 3], #arrA
[2, 3, 4], #arrB
[4, 5, 6]] #arrC
distance_matrix = np.array([])
for target_array in array_list:
print (target_array)
print (array_list)
#row should be an array of jac distances between target_array and each array in array_list
row = vectorised_compute_jac(target_array , array_list)
print (row, '\n\n')
#np.vectorise() functions return an array of objects of type specified by otype param, based on docs
np.append(distance_matrix, row)
Output + Error:
[1, 2, 3]
[[1, 2, 3], [2, 3, 4], [4, 5, 6]]
Traceback (most recent call last):
File "C:\Users\u03132tk\.spyder-py3\ModuleMapper\untitled1.py", line 21, in <module>
row = vectorised_compute_jac(array, array_list)
File "C:\ANACONDA3\lib\site-packages\numpy\lib\function_base.py", line 2163, in __call__
return self._vectorize_call(func=func, args=vargs)
File "C:\ANACONDA3\lib\site-packages\numpy\lib\function_base.py", line 2237, in _vectorize_call
res = self._vectorize_call_with_signature(func, args)
File "C:\ANACONDA3\lib\site-packages\numpy\lib\function_base.py", line 2277, in _vectorize_call_with_signature
results = func(*(arg[index] for arg in args))
File "C:\ANACONDA3\lib\site-packages\scipy\spatial\distance.py", line 893, in jaccard
v = _validate_vector(v)
File "C:\ANACONDA3\lib\site-packages\scipy\spatial\distance.py", line 340, in _validate_vector
raise ValueError("Input vector should be 1-D.")
ValueError: Input vector should be 1-D.
What I would like, with square brackets indicating numpy arrays not lists, based on array output types discussed in comments above:
#arrA #arrB #arrC
[[JD(AA), JD(AB), JD(AC)], #arrA
[JD(BA), JD(BB), JD(BC)], #arrB
[JD(CA), JD(CB), JD(CC)]] #arrC
Can someone advise how the signature param works and whether thats causing my woes? I suspect it's due to the (n, m) in my signature as it's the only multi-dimensional thing, hence the question :(
Cheers!
Tim
I was going to run your code as is, but then saw that you were misusing np.append. So I'll skip your iteration, and try to recreate the calculation with straight forward list comprehensions.
It looks like jaccard takes 2 1d arrays, and returns a scalar, and you apparently want to calculate it for all pairs of your list of arrays.
In [5]: arr = np.array(array_list)
In [6]: [jaccard(arr[0],b) for b in arr]
Out[6]: [0.0, 1.0, 1.0]
In [7]: [[jaccard(a,b) for b in arr] for a in arr]
Out[7]: [[0.0, 1.0, 1.0], [1.0, 0.0, 1.0], [1.0, 1.0, 0.0]]
In [9]: np.array(_)
Out[9]:
array([[0., 1., 1.],
[1., 0., 1.],
[1., 1., 0.]])
With symmetry and 0s it should be possible to cut down on the jaccard calls with a more selective iteration. But I'll leave that for others.
With your signature, you are telling vectorize to pass a 1d and a 2d array to the jaccard, and to expect back a 1d. That's not right.
This is, I think the correct use of vectorize:
In [12]: vectorised_compute_jac = np.vectorize(jaccard, signature = '(m),(m)->()
...: ')
In [13]: vectorised_compute_jac(arr[None,:,:],arr[:,None,:])
Out[13]:
array([[0., 1., 1.],
[1., 0., 1.],
[1., 1., 0.]])
Compare its time against the nested comprehension:
In [14]: timeit vectorised_compute_jac(arr[None,:,:],arr[:,None,:])
384 µs ± 5.34 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [15]: timeit np.array([[jaccard(a,b) for b in arr] for a in arr])
203 µs ± 204 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [15], it's the jaccard calls that dominate the time, not the iteration mechanism. So taking advantage of the symmetry will worth it.
In [17]: timeit jaccard(arr[0],arr[1])
21.2 µs ± 79.5 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
I have an array of values arr with shape (N,) and an array of coordinates coords with shape (N,2). I want to represent this in an (M,M) array grid such that grid takes the value 0 at coordinates that are not in coords, and for the coordinates that are included it should store the sum of all values in arr that have that coordinate. So if M=3, arr = np.arange(4)+1, and coords = np.array([[0,0,1,2],[0,0,2,2]]) then grid should be:
array([[3., 0., 0.],
[0., 0., 3.],
[0., 0., 4.]])
The reason this is nontrivial is that I need to be able to repeat this step many times and the values in arr change each time, and so can the coordinates. Ideally I am looking for a vectorized solution. I suspect that I might be able to use np.where somehow but it's not immediately obvious how.
Timing the solutions
I have timed the solutions present at this time and it appear that the accumulator method is slightly faster than the sparse matrix method, with the second accumulation method being the slowest for the reasons explained in the comments:
%timeit for x in range(100): accumulate_arr(np.random.randint(100,size=(2,10000)),np.random.normal(0,1,10000))
%timeit for x in range(100): accumulate_arr_v2(np.random.randint(100,size=(2,10000)),np.random.normal(0,1,10000))
%timeit for x in range(100): sparse.coo_matrix((np.random.normal(0,1,10000),np.random.randint(100,size=(2,10000))),(100,100)).A
47.3 ms ± 1.79 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
103 ms ± 255 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
48.2 ms ± 36 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
One way would be to create a sparse.coo_matrix and convert that to dense:
from scipy import sparse
sparse.coo_matrix((arr,coords),(M,M)).A
# array([[3, 0, 0],
# [0, 0, 3],
# [0, 0, 4]])
With np.bincount -
def accumulate_arr(coords, arr):
# Get output array shape
m,n = coords.max(1)+1
# Get linear indices to be used as IDs with bincount
lidx = np.ravel_multi_index(coords, (m,n))
# Or lidx = coords[0]*(coords[1].max()+1) + coords[1]
# Accumulate arr with IDs from lidx
return np.bincount(lidx,arr,minlength=m*n).reshape(m,n)
Sample run -
In [58]: arr
Out[58]: array([1, 2, 3, 4])
In [59]: coords
Out[59]:
array([[0, 0, 1, 2],
[0, 0, 2, 2]])
In [60]: accumulate_arr(coords, arr)
Out[60]:
array([[3., 0., 0.],
[0., 0., 3.],
[0., 0., 4.]])
Another with np.add.at on similar lines and might be easier to follow -
def accumulate_arr_v2(coords, arr):
m,n = coords.max(1)+1
out = np.zeros((m,n), dtype=arr.dtype)
np.add.at(out, tuple(coords), arr)
return out
This question already has answers here:
NumPy Broadcasting: Calculating sum of squared differences between two arrays
(3 answers)
Closed 4 years ago.
Basically, I have two matrices A and B, and I want C (dimensions marked by the side of the matrices), with computation like this:
The formula below is what I do now. I take advantage of some broadcasting, but I am still left with a loop. I am novel to Python so maybe I am wrong, but I just have a hunch that this loop can be eliminated. Can anyone share some ideas?
EDIT: 2018-04-27 09:48:28
as requested, an example:
In [5]: A
Out[5]:
array([[0, 1],
[2, 3],
[4, 5],
[6, 7],
[8, 9]])
In [6]: B
Out[6]:
array([[0, 1],
[2, 3],
[4, 5],
[6, 7]])
In [7]: C = np.zeros ((B.shape[0], A.shape[0]))
In [8]: for m in range (B.shape[0]):
...: C[m] = np.sum (np.square (B[m] - A), axis=1).flatten ()
...:
In [9]: C
Out[9]:
array([[ 0., 8., 32., 72., 128.],
[ 8., 0., 8., 32., 72.],
[ 32., 8., 0., 8., 32.],
[ 72., 32., 8., 0., 8.]])
This appears to work at the cost of some extra memory:
C = ((B[:, :, None] - A.T)**2).sum(axis=1)
Testing:
import numpy
D = 10
N = 20
M = 30
A = numpy.random.rand(N, D)
B = numpy.random.rand(M, D)
C = numpy.empty((M, N))
Timing:
for m in range(M):
C[m] = numpy.sum((B[m, :] - A)**2, axis=1)
514 µs ± 13.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
C2 = ((B[:, :, None] - A.T)**2).sum(axis=1)
53.6 µs ± 529 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
I'm searching for an efficient way to create a matrix of occurrences from two arrays that contains indexes, one represents the row indexes in this matrix, the other, the column ones.
eg. I have:
#matrix will be size 4x3 in this example
#array of rows idxs, with values from 0 to 3
[0, 1, 1, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3]
#array of columns idxs, with values from 0 to 2
[0, 1, 1, 1, 2, 2, 0, 1, 2, 0, 2, 2, 2, 2]
And need to create a matrix of occurrences like:
[[1 0 0]
[0 2 0]
[0 1 2]
[2 1 5]]
I can create an array of one hot vectors in a simple form, but cant get it work when there is more than one occurrence:
n_rows = 4
n_columns = 3
#data
rows = np.array([0, 1, 1, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3])
columns = np.array([0, 1, 1, 1, 2, 2, 0, 1, 2, 0, 2, 2, 2, 2])
#empty matrix
new_matrix = np.zeros([n_rows, n_columns])
#adding 1 for each [row, column] occurrence:
new_matrix[rows, columns] += 1
print(new_matrix)
Which returns:
[[ 1. 0. 0.]
[ 0. 1. 0.]
[ 0. 1. 1.]
[ 1. 1. 1.]]
It seems like indexing and adding a value like this doesn't work when there is more than one occurrence/index, besides printing it seems to work just fine:
print(new_matrix[rows, :])
:
[[ 1. 0. 0.]
[ 0. 1. 0.]
[ 0. 1. 0.]
[ 0. 1. 1.]
[ 0. 1. 1.]
[ 0. 1. 1.]
[ 1. 1. 1.]
[ 1. 1. 1.]
[ 1. 1. 1.]
[ 1. 1. 1.]
[ 1. 1. 1.]
[ 1. 1. 1.]
[ 1. 1. 1.]
[ 1. 1. 1.]]
So maybe I'm missing something there? Or this cant be done and I need to search for another way to do it?
Use np.add.at, specifying a tuple of indices:
>>> np.add.at(new_matrix, (rows, columns), 1)
>>> new_matrix
array([[ 1., 0., 0.],
[ 0., 2., 0.],
[ 0., 1., 2.],
[ 2., 1., 5.]])
np.add.at operates on the array in-place, adding 1 as many times to the indices as specified by the (row, columns) tuple.
Approach #1
We can convert those pairs to linear indices and then use np.bincount -
def bincount_app(rows, columns, n_rows, n_columns):
# Get linear index equivalent
lidx = (columns.max()+1)*rows + columns
# Use binned count on the linear indices
return np.bincount(lidx, minlength=n_rows*n_columns).reshape(n_rows,n_columns)
Sample run -
In [242]: n_rows = 4
...: n_columns = 3
...:
...: rows = np.array([0, 1, 1, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3])
...: columns = np.array([0, 1, 1, 1, 2, 2, 0, 1, 2, 0, 2, 2, 2, 2])
In [243]: bincount_app(rows, columns, n_rows, n_columns)
Out[243]:
array([[1, 0, 0],
[0, 2, 0],
[0, 1, 2],
[2, 1, 5]])
Approach #2
Alternatively, we can sort the linear indices and get the counts using slicing to have our second approach, like so -
def mask_diff_app(rows, columns, n_rows, n_columns):
lidx = (columns.max()+1)*rows + columns
lidx.sort()
mask = np.concatenate(([True],lidx[1:] != lidx[:-1],[True]))
count = np.diff(np.flatnonzero(mask))
new_matrix = np.zeros([n_rows, n_columns],dtype=int)
new_matrix.flat[lidx[mask[:-1]]] = count
return new_matrix
Approach #3
This seems like a straight-forward one with sparse matrix csr_matrix as well, as it does accumulation on its own for repeated indices. The benefit is the memory efficiency, given that it's a sparse matrix, which would be noticeable if you are filling a small number of places in the output and a sparse matrix output is okay.
The implementation would look something like this -
from scipy.sparse import csr_matrix
def sparse_matrix_app(rows, columns, n_rows, n_columns):
out_shp = (n_rows, n_columns)
data = np.ones(len(rows),dtype=int)
return csr_matrix((data, (rows, columns)), shape=out_shp)
If you need a regular/dense array, simply do -
sparse_matrix_app(rows, columns, n_rows, n_columns).toarray()
Sample output -
In [319]: sparse_matrix_app(rows, columns, n_rows, n_columns).toarray()
Out[319]:
array([[1, 0, 0],
[0, 2, 0],
[0, 1, 2],
[2, 1, 5]])
Benchmarking
Other approach(es) -
# #cᴏʟᴅsᴘᴇᴇᴅ's soln
def add_at_app(rows, columns, n_rows, n_columns):
new_matrix = np.zeros([n_rows, n_columns],dtype=int)
np.add.at(new_matrix, (rows, columns), 1)
Timings
Case #1 : Output array of shape (1000, 1000) and no. of indices = 10k
In [307]: # Setup
...: n_rows = 1000
...: n_columns = 1000
...: rows = np.random.randint(0,1000,(10000))
...: columns = np.random.randint(0,1000,(10000))
In [308]: %timeit add_at_app(rows, columns, n_rows, n_columns)
...: %timeit bincount_app(rows, columns, n_rows, n_columns)
...: %timeit mask_diff_app(rows, columns, n_rows, n_columns)
...: %timeit sparse_matrix_app(rows, columns, n_rows, n_columns)
1000 loops, best of 3: 1.05 ms per loop
1000 loops, best of 3: 424 µs per loop
1000 loops, best of 3: 1.05 ms per loop
1000 loops, best of 3: 1.41 ms per loop
Case #2 : Output array of shape (1000, 1000) and no. of indices = 100k
In [309]: # Setup
...: n_rows = 1000
...: n_columns = 1000
...: rows = np.random.randint(0,1000,(100000))
...: columns = np.random.randint(0,1000,(100000))
In [310]: %timeit add_at_app(rows, columns, n_rows, n_columns)
...: %timeit bincount_app(rows, columns, n_rows, n_columns)
...: %timeit mask_diff_app(rows, columns, n_rows, n_columns)
...: %timeit sparse_matrix_app(rows, columns, n_rows, n_columns)
100 loops, best of 3: 11.4 ms per loop
1000 loops, best of 3: 1.27 ms per loop
100 loops, best of 3: 7.44 ms per loop
10 loops, best of 3: 20.4 ms per loop
Case #3 : Sparse-ness in output
As stated earlier, for the sparse method to work better, we would need sparse-ness. Such a case would be like this -
In [314]: # Setup
...: n_rows = 5000
...: n_columns = 5000
...: rows = np.random.randint(0,5000,(1000))
...: columns = np.random.randint(0,5000,(1000))
In [315]: %timeit add_at_app(rows, columns, n_rows, n_columns)
...: %timeit bincount_app(rows, columns, n_rows, n_columns)
...: %timeit mask_diff_app(rows, columns, n_rows, n_columns)
...: %timeit sparse_matrix_app(rows, columns, n_rows, n_columns)
100 loops, best of 3: 11.7 ms per loop
100 loops, best of 3: 11.1 ms per loop
100 loops, best of 3: 11.1 ms per loop
1000 loops, best of 3: 269 µs per loop
If you need a dense array, we lose the memory efficiency and hence performance one as well -
In [317]: %timeit sparse_matrix_app(rows, columns, n_rows, n_columns).toarray()
100 loops, best of 3: 11.7 ms per loop
Sometimes it is useful to "clone" a row or column vector to a matrix. By cloning I mean converting a row vector such as
[1, 2, 3]
Into a matrix
[[1, 2, 3],
[1, 2, 3],
[1, 2, 3]]
or a column vector such as
[[1],
[2],
[3]]
into
[[1, 1, 1]
[2, 2, 2]
[3, 3, 3]]
In MATLAB or octave this is done pretty easily:
x = [1, 2, 3]
a = ones(3, 1) * x
a =
1 2 3
1 2 3
1 2 3
b = (x') * ones(1, 3)
b =
1 1 1
2 2 2
3 3 3
I want to repeat this in numpy, but unsuccessfully
In [14]: x = array([1, 2, 3])
In [14]: ones((3, 1)) * x
Out[14]:
array([[ 1., 2., 3.],
[ 1., 2., 3.],
[ 1., 2., 3.]])
# so far so good
In [16]: x.transpose() * ones((1, 3))
Out[16]: array([[ 1., 2., 3.]])
# DAMN
# I end up with
In [17]: (ones((3, 1)) * x).transpose()
Out[17]:
array([[ 1., 1., 1.],
[ 2., 2., 2.],
[ 3., 3., 3.]])
Why wasn't the first method (In [16]) working? Is there a way to achieve this task in python in a more elegant way?
Use numpy.tile:
>>> tile(array([1,2,3]), (3, 1))
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
or for repeating columns:
>>> tile(array([[1,2,3]]).transpose(), (1, 3))
array([[1, 1, 1],
[2, 2, 2],
[3, 3, 3]])
Here's an elegant, Pythonic way to do it:
>>> array([[1,2,3],]*3)
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
>>> array([[1,2,3],]*3).transpose()
array([[1, 1, 1],
[2, 2, 2],
[3, 3, 3]])
the problem with [16] seems to be that the transpose has no effect for an array. you're probably wanting a matrix instead:
>>> x = array([1,2,3])
>>> x
array([1, 2, 3])
>>> x.transpose()
array([1, 2, 3])
>>> matrix([1,2,3])
matrix([[1, 2, 3]])
>>> matrix([1,2,3]).transpose()
matrix([[1],
[2],
[3]])
First note that with numpy's broadcasting operations it's usually not necessary to duplicate rows and columns. See this and this for descriptions.
But to do this, repeat and newaxis are probably the best way
In [12]: x = array([1,2,3])
In [13]: repeat(x[:,newaxis], 3, 1)
Out[13]:
array([[1, 1, 1],
[2, 2, 2],
[3, 3, 3]])
In [14]: repeat(x[newaxis,:], 3, 0)
Out[14]:
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
This example is for a row vector, but applying this to a column vector is hopefully obvious. repeat seems to spell this well, but you can also do it via multiplication as in your example
In [15]: x = array([[1, 2, 3]]) # note the double brackets
In [16]: (ones((3,1))*x).transpose()
Out[16]:
array([[ 1., 1., 1.],
[ 2., 2., 2.],
[ 3., 3., 3.]])
Let:
>>> n = 1000
>>> x = np.arange(n)
>>> reps = 10000
Zero-cost allocations
A view does not take any additional memory. Thus, these declarations are instantaneous:
# New axis
x[np.newaxis, ...]
# Broadcast to specific shape
np.broadcast_to(x, (reps, n))
Forced allocation
If you want force the contents to reside in memory:
>>> %timeit np.array(np.broadcast_to(x, (reps, n)))
10.2 ms ± 62.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> %timeit np.repeat(x[np.newaxis, :], reps, axis=0)
9.88 ms ± 52.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> %timeit np.tile(x, (reps, 1))
9.97 ms ± 77.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
All three methods are roughly the same speed.
Computation
>>> a = np.arange(reps * n).reshape(reps, n)
>>> x_tiled = np.tile(x, (reps, 1))
>>> %timeit np.broadcast_to(x, (reps, n)) * a
17.1 ms ± 284 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> %timeit x[np.newaxis, :] * a
17.5 ms ± 300 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> %timeit x_tiled * a
17.6 ms ± 240 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
All three methods are roughly the same speed.
Conclusion
If you want to replicate before a computation, consider using one of the "zero-cost allocation" methods. You won't suffer the performance penalty of "forced allocation".
I think using the broadcast in numpy is the best, and faster
I did a compare as following
import numpy as np
b = np.random.randn(1000)
In [105]: %timeit c = np.tile(b[:, newaxis], (1,100))
1000 loops, best of 3: 354 µs per loop
In [106]: %timeit c = np.repeat(b[:, newaxis], 100, axis=1)
1000 loops, best of 3: 347 µs per loop
In [107]: %timeit c = np.array([b,]*100).transpose()
100 loops, best of 3: 5.56 ms per loop
about 15 times faster using broadcast
One clean solution is to use NumPy's outer-product function with a vector of ones:
np.outer(np.ones(n), x)
gives n repeating rows. Switch the argument order to get repeating columns. To get an equal number of rows and columns you might do
np.outer(np.ones_like(x), x)
You can use
np.tile(x,3).reshape((4,3))
tile will generate the reps of the vector
and reshape will give it the shape you want
Returning to the original question
In MATLAB or octave this is done pretty easily:
x = [1, 2, 3]
a = ones(3, 1) * x
...
In numpy it's pretty much the same (and easy to memorize too):
x = [1, 2, 3]
a = np.tile(x, (3, 1))
If you have a pandas dataframe and want to preserve the dtypes, even the categoricals, this is a fast way to do it:
import numpy as np
import pandas as pd
df = pd.DataFrame({1: [1, 2, 3], 2: [4, 5, 6]})
number_repeats = 50
new_df = df.reindex(np.tile(df.index, number_repeats))
Another solution
>> x = np.array([1,2,3])
>> y = x[None, :] * np.ones((3,))[:, None]
>> y
array([[ 1., 2., 3.],
[ 1., 2., 3.],
[ 1., 2., 3.]])
Why? Sure, repeat and tile are the correct way to do this. But None indexing is a powerful tool that has many times let me quickly vectorize an operation (though it can quickly be very memory expensive!).
An example from my own code:
# trajectory is a sequence of xy coordinates [n_points, 2]
# xy_obstacles is a list of obstacles' xy coordinates [n_obstacles, 2]
# to compute dx, dy distance between every obstacle and every pose in the trajectory
deltas = trajectory[:, None, :2] - xy_obstacles[None, :, :2]
# we can easily convert x-y distance to a norm
distances = np.linalg.norm(deltas, axis=-1)
# distances is now [timesteps, obstacles]. Now we can for example find the closest obstacle at every point in the trajectory by doing
closest_obstacles = np.argmin(distances, axis=1)
# we could also find how safe the trajectory is, by finding the smallest distance over the entire trajectory
danger = np.min(distances)
To answer the actual question, now that nearly a dozen approaches to working around a solution have been posted: x.transpose reverses the shape of x. One of the interesting side-effects is that if x.ndim == 1, the transpose does nothing.
This is especially confusing for people coming from MATLAB, where all arrays implicitly have at least two dimensions. The correct way to transpose a 1D numpy array is not x.transpose() or x.T, but rather
x[:, None]
or
x.reshape(-1, 1)
From here, you can multiply by a matrix of ones, or use any of the other suggested approaches, as long as you respect the (subtle) differences between MATLAB and numpy.
import numpy as np
x=np.array([1,2,3])
y=np.multiply(np.ones((len(x),len(x))),x).T
print(y)
yields:
[[ 1. 1. 1.]
[ 2. 2. 2.]
[ 3. 3. 3.]]