Related
Given python code with numpy:
import numpy as np
a = np.arange(6).reshape(3, 2) # a = [[0, 1], [2, 3], [4, 5]]; a.shape = (3, 2)
b = np.arange(3) + 1 # b = [1, 2, 3] ; b.shape = (3,)
How can I multiply each value in b with each corresponding row ('vector') in a? So here, I want the result as:
result = [[0, 1], [4, 6], [12, 15]] # result.shape = (3, 2)
I can do this with a loop, but I am wondering about a vectorized approach. I found an Octave solution here. Apart from this, I didn't find anything else. Any pointers for this?
Thank you in advance.
Probably the simplest is to do the following.
import numpy as np
a = np.arange(6).reshape(3, 2) # a = [[0, 1], [2, 3], [4, 5]]; a.shape = (3, 2)
b = np.arange(3) + 1
ans = np.diag(b)#a
Here's a method that exploits numpy multiplication broadcasting:
ans = (b*a.T).T
These two solutions basically take the same approach
ans = np.tile(b,(2,1)).T*a
ans = np.vstack([b for _ in range(a.shape[1])]).T*a
In [123]: a = np.arange(6).reshape(3, 2) # a = [[0, 1], [2, 3], [4, 5]]; a.
...: shape = (3, 2)
...: b = np.arange(3) + 1 # b = [1, 2, 3] ; b.
...: shape = (3,)
In [124]: a
Out[124]:
array([[0, 1],
[2, 3],
[4, 5]])
A (3,1) will multiply a (3,2) via broadcasting:
In [125]: a*b[:,None]
Out[125]:
array([[ 0, 1],
[ 4, 6],
[12, 15]])
I have a NumPy array with each row representing some (x, y, z) coordinate like so:
a = array([[0, 0, 1],
[1, 1, 2],
[4, 5, 1],
[4, 5, 2]])
I also have another NumPy array with unique values of the z-coordinates of that array like so:
b = array([1, 2])
How can I apply a function, let's call it "f", to each of the groups of rows in a which correspond to the values in b? For example, the first value of b is 1 so I would get all rows of a which have a 1 in the z-coordinate. Then, I apply a function to all those values.
In the end, the output would be an array the same shape as b.
I'm trying to vectorize this to make it as fast as possible. Thanks!
Example of an expected output (assuming that f is count()):
c = array([2, 2])
because there are 2 rows in array a which have a z value of 1 in array b and also 2 rows in array a which have a z value of 2 in array b.
A trivial solution would be to iterate over array b like so:
for val in b:
apply function to a based on val
append to an array c
My attempt:
I tried doing something like this, but it just returns an empty array.
func(a[a[:, 2]==b])
The problem is that the groups of rows with the same Z can have different sizes so you cannot stack them into one 3D numpy array which would allow to easily apply a function along the third dimension. One solution is to use a for-loop, another is to use np.split:
a = np.array([[0, 0, 1],
[1, 1, 2],
[4, 5, 1],
[4, 5, 2],
[4, 3, 1]])
a_sorted = a[a[:,2].argsort()]
inds = np.unique(a_sorted[:,2], return_index=True)[1]
a_split = np.split(a_sorted, inds)[1:]
# [array([[0, 0, 1],
# [4, 5, 1],
# [4, 3, 1]]),
# array([[1, 1, 2],
# [4, 5, 2]])]
f = np.sum # example of a function
result = list(map(f, a_split))
# [19, 15]
But imho the best solution is to use pandas and groupby as suggested by FBruzzesi. You can then convert the result to a numpy array.
EDIT: For completeness, here are the other two solutions
List comprehension:
b = np.unique(a[:,2])
result = [f(a[a[:,2] == z]) for z in b]
Pandas:
df = pd.DataFrame(a, columns=list('XYZ'))
result = df.groupby(['Z']).apply(lambda x: f(x.values)).tolist()
This is the performance plot I got for a = np.random.randint(0, 100, (n, 3)):
As you can see, approximately up to n = 10^5 the "split solution" is the fastest, but after that the pandas solution performs better.
If you are allowed to use pandas:
import pandas as pd
df=pd.DataFrame(a, columns=['x','y','z'])
df.groupby('z').agg(f)
Here f can be any custom function working on grouped data.
Numeric example:
a = np.array([[0, 0, 1],
[1, 1, 2],
[4, 5, 1],
[4, 5, 2]])
df=pd.DataFrame(a, columns=['x','y','z'])
df.groupby('z').size()
z
1 2
2 2
dtype: int64
Remark that .size is the way to count number of rows per group.
To keep it into pure numpy, maybe this can suit your case:
tmp = np.array([a[a[:,2]==i] for i in b])
tmp
array([[[0, 0, 1],
[4, 5, 1]],
[[1, 1, 2],
[4, 5, 2]]])
which is an array with each group of arrays.
c = np.array([])
for x in np.nditer(b):
c = np.append(c, np.where((a[:,2] == x))[0].shape[0])
Output:
[2. 2.]
I have an array like this and need to replace every 1 with 2, every 3 with 4, every 4 with 1. Is there a way to do this just with np and not loops?
import numpy as np
np.random.seed(2)
arr=np.random.randint(1,5,(3,3),int)
arr
array([[1, 4, 2],
[1, 3, 4],
[3, 4, 1]])
If I use array mask sequentially, it doesn't give the expected outcome:
array([[2, 1, 2],
[2, 4, 1],
[4, 1, 2]])
It is based on a conditional logic and not maths formula
If the array values don't necessarely range between 1 and 4 you can use np.select:
import numpy as np
a = np.random.randint(1,5, (3,3))
condlist = [np.logical_or(a==1, a==2), a==3, a==4]
choicelist= [2, 4, 1]
b = np.select(condlist, choicelist)
which does not care about the order of the conditions
Here's one with np.searchsorted for performance efficiency -
def map_values(arr, old_val, new_val):
sidx = old_val.argsort()
idx = np.searchsorted(old_val,arr,sorter=sidx)
return np.where(old_val[idx]==arr, new_val[sidx[idx]], arr)
Sample run -
In [40]: arr
Out[40]:
array([[1, 4, 2],
[1, 3, 4],
[3, 4, 1]])
In [41]: old_val = np.array([1,3,4])
...: new_val = np.array([2,4,1])
In [42]: map_values(arr, old_val, new_val)
Out[42]:
array([[2, 1, 2],
[2, 4, 1],
[4, 1, 2]])
Could do this with a lambda function and np.vectorize():
import numpy as np
np.random.seed(2)
arr=np.random.randint(1,5,(3,3),int)
f = lambda x: x%4 + 1 if x in [1,3,4] else x
vfunc = np.vectorize(f)
Usage:
>>> vfunc(arr)
array([[2, 1, 2],
[2, 4, 1],
[4, 1, 2]])
You have to be careful about the order of assignments. For example, if you do
arr[arr == 4] = 1
arr[arr == 1] = 2
Now all of the elements that were originally 4 will be 2, not 1 as you intend.
One solution is to carefully craft the order of assignments:
arr[arr == 1] = 2
arr[arr == 4] = 1
However, this is very brittle and will fall apart as you introduce more of them. It would be better to create the masks up front from the original array:
ones = arr == 1
fours = arr == 4
arr[ones] = 2
arr[fours] = 1
Now the order of the assignments won't matter because the masks are determined before modifying the array.
You want arr % 4 + 1, except in the case of 2, which stays the same. So use np.where to find all the 2s. Then do arr % 4 + 1, then reset all the 2s.
import numpy as np
np.random.seed(2)
arr=np.random.randint(1,5,(3,3),int)
twos = np.where(arr == 2)
arr = arr % 4 + 1
arr[twos] = 2
print(arr)
I am trying to replace one or several columns with a new array with the same length.
a = np.array([[1,2,3],[1,2,3],[1,2,3]])
b = np.array([[0,0,0])
a[:, 0] = b
I got an error of ValueError: could not broadcast input array from shape (3,1) into shape (3). However this works when b has multiple columns.
a = np.array([[1,2,3],[1,2,3],[1,2,3]])
b = np.array([[0,7],[0,7],[0,7]])
a[:, 0:2] = b
array([[0, 7, 3],
[0, 7, 3],
[0, 7, 3]])
How can I efficiently replace a column with another array?
Thanks
J
Your example will work fine if you use the following just like you are using a[:, 0:2] = b. [:, 0:1] is effectively just the first column
a = np.array([[1,2,3],[1,2,3],[1,2,3]])
b = np.array([[0],[0],[0]])
a[:, 0:1] = b
# array([[0, 2, 3],
# [0, 2, 3],
# [0, 2, 3]])
You have an incorrect shape of b. You should pass an ordinary 1D array to it if you want to replace only one column:
a = np.array([[1,2,3],[1,2,3],[1,2,3]])
b = np.array([0,0,0])
a[:, 0] = b
a
Returns:
array([[0, 2, 3],
[0, 2, 3],
[0, 2, 3]])
I'd like to roll a 2D numpy in python, except that I'd like pad the ends with zeros rather than roll the data as if its periodic.
Specifically, the following code
import numpy as np
x = np.array([[1, 2, 3], [4, 5, 6]])
np.roll(x, 1, axis=1)
returns
array([[3, 1, 2],[6, 4, 5]])
but what I would prefer is
array([[0, 1, 2], [0, 4, 5]])
I could do this with a few awkward touchups, but I'm hoping that there's a way to do it with fast built-in commands.
Thanks
There is a new numpy function in version 1.7.0 numpy.pad that can do this in one-line. Pad seems to be quite powerful and can do much more than a simple "roll". The tuple ((0,0),(1,0)) used in this answer indicates the "side" of the matrix which to pad.
import numpy as np
x = np.array([[1, 2, 3],[4, 5, 6]])
print np.pad(x,((0,0),(1,0)), mode='constant')[:, :-1]
Giving
[[0 1 2]
[0 4 5]]
I don't think that you are going to find an easier way to do this that is built-in. The touch-up seems quite simple to me:
y = np.roll(x,1,axis=1)
y[:,0] = 0
If you want this to be more direct then maybe you could copy the roll function to a new function and change it to do what you want. The roll() function is in the site-packages\core\numeric.py file.
I just wrote the following. It could be more optimized by avoiding zeros_like and just computing the shape for zeros directly.
import numpy as np
def roll_zeropad(a, shift, axis=None):
"""
Roll array elements along a given axis.
Elements off the end of the array are treated as zeros.
Parameters
----------
a : array_like
Input array.
shift : int
The number of places by which elements are shifted.
axis : int, optional
The axis along which elements are shifted. By default, the array
is flattened before shifting, after which the original
shape is restored.
Returns
-------
res : ndarray
Output array, with the same shape as `a`.
See Also
--------
roll : Elements that roll off one end come back on the other.
rollaxis : Roll the specified axis backwards, until it lies in a
given position.
Examples
--------
>>> x = np.arange(10)
>>> roll_zeropad(x, 2)
array([0, 0, 0, 1, 2, 3, 4, 5, 6, 7])
>>> roll_zeropad(x, -2)
array([2, 3, 4, 5, 6, 7, 8, 9, 0, 0])
>>> x2 = np.reshape(x, (2,5))
>>> x2
array([[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9]])
>>> roll_zeropad(x2, 1)
array([[0, 0, 1, 2, 3],
[4, 5, 6, 7, 8]])
>>> roll_zeropad(x2, -2)
array([[2, 3, 4, 5, 6],
[7, 8, 9, 0, 0]])
>>> roll_zeropad(x2, 1, axis=0)
array([[0, 0, 0, 0, 0],
[0, 1, 2, 3, 4]])
>>> roll_zeropad(x2, -1, axis=0)
array([[5, 6, 7, 8, 9],
[0, 0, 0, 0, 0]])
>>> roll_zeropad(x2, 1, axis=1)
array([[0, 0, 1, 2, 3],
[0, 5, 6, 7, 8]])
>>> roll_zeropad(x2, -2, axis=1)
array([[2, 3, 4, 0, 0],
[7, 8, 9, 0, 0]])
>>> roll_zeropad(x2, 50)
array([[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]])
>>> roll_zeropad(x2, -50)
array([[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]])
>>> roll_zeropad(x2, 0)
array([[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9]])
"""
a = np.asanyarray(a)
if shift == 0: return a
if axis is None:
n = a.size
reshape = True
else:
n = a.shape[axis]
reshape = False
if np.abs(shift) > n:
res = np.zeros_like(a)
elif shift < 0:
shift += n
zeros = np.zeros_like(a.take(np.arange(n-shift), axis))
res = np.concatenate((a.take(np.arange(n-shift,n), axis), zeros), axis)
else:
zeros = np.zeros_like(a.take(np.arange(n-shift,n), axis))
res = np.concatenate((zeros, a.take(np.arange(n-shift), axis)), axis)
if reshape:
return res.reshape(a.shape)
else:
return res
import numpy as np
def shift_2d_replace(data, dx, dy, constant=False):
"""
Shifts the array in two dimensions while setting rolled values to constant
:param data: The 2d numpy array to be shifted
:param dx: The shift in x
:param dy: The shift in y
:param constant: The constant to replace rolled values with
:return: The shifted array with "constant" where roll occurs
"""
shifted_data = np.roll(data, dx, axis=1)
if dx < 0:
shifted_data[:, dx:] = constant
elif dx > 0:
shifted_data[:, 0:dx] = constant
shifted_data = np.roll(shifted_data, dy, axis=0)
if dy < 0:
shifted_data[dy:, :] = constant
elif dy > 0:
shifted_data[0:dy, :] = constant
return shifted_data
This function would work on 2D arrays and replace rolled values with a constant of your choosing.
A bit late, but feels like a quick way to do what you want in one line. Perhaps would work best if wrapped inside a smart function (example below provided just for horizontal axis):
import numpy
a = numpy.arange(1,10).reshape(3,3) # an example 2D array
print a
[[1 2 3]
[4 5 6]
[7 8 9]]
shift = 1
a = numpy.hstack((numpy.zeros((a.shape[0], shift)), a[:,:-shift]))
print a
[[0 1 2]
[0 4 5]
[0 7 8]]
You can also use ndimage.shift:
>>> from scipy import ndimage
>>> arr = np.array([[1, 2, 3], [4, 5, 6]])
>>> ndimage.shift(arr, (0,1))
array([[0, 1, 2],
[0, 4, 5]])
Elaborating on the answer by Hooked (since it took me a few minutes to understand it)
The code below first pads a certain amount of zeros in the up, down, left and right margins and then selects the original matrix inside the padded one. A perfectly useless code, but good for understanding np.pad.
import numpy as np
x = np.array([[1, 2, 3],[4, 5, 6]])
y = np.pad(x,((1,3),(2,4)), mode='constant')[1:-3,2:-4]
print np.all(x==y)
now to make an upwards shift of 2 combined with a rightwards shift of 1 position one should do
print np.pad(x,((0,2),(1,0)), mode='constant')[2:0,0:-1]
You could also use numpy's triu and scipy.linalg's circulant. Make a circulant version of your matrix. Then, select the upper triangular part starting at the first diagonal, (the default option in triu). The row index will correspond to the number of padded zeros you want.
If you don't have scipy you can generate a nXn circulant matrix by making an (n-1) X (n-1) identity matrix and stacking a row [0 0 ... 1] on top of it and the column [1 0 ... 0] to the right of it.
I faced a similar problem with shifting a 2-d array in both directions
def shift_frame(img,move_dir,fill=np.inf):
frame = np.full_like(img,fill)
x,y = move_dir
size_x,size_y = np.array(img.shape) - np.abs(move_dir)
frame_x = slice(0,size_x) if x>=0 else slice(-x,size_x-x)
frame_y = slice(0,size_y) if y>=0 else slice(-y,size_y-y)
img_x = slice(x,None) if x>=0 else slice(0,size_x)
img_y = slice(y,None) if y>=0 else slice(0,size_y)
frame[frame_x,frame_y] = img[img_x,img_y]
return frame
test = np.arange(25).reshape((5,5))
shift_frame(test,(1,1))
'''
returns:
array([[ 6, 7, 8, 9, -1],
[11, 12, 13, 14, -1],
[16, 17, 18, 19, -1],
[21, 22, 23, 24, -1],
[-1, -1, -1, -1, -1]])
'''
I haven't measured the runtime of this, but it seems to work well enough for my use, although a built-in one liner would be nice
import numpy as np
def roll_zeropad(a, dyx):
h, w = a.shape[:2]
dy, dx = dyx
pad_x, start_x, end_x = ((dx,0), 0, w) if dx > 0 else ((0,-dx), -dx, w-dx)
pad_y, start_y, end_y = ((dy,0), 0, h) if dy > 0 else ((0,-dy), -dy, h-dy)
return np.pad(a, (pad_y, pad_x))[start_y:end_y,start_x:end_x]
test = np.arange(25).reshape((5,5))
out = roll_zeropad(test,(1,1))
print(out)
"""
returns:
[[ 0 0 0 0 0]
[ 0 0 1 2 3]
[ 0 5 6 7 8]
[ 0 10 11 12 13]
[ 0 15 16 17 18]]
"""