Develop Reference

Vectorize list returning python function into numpy nd-array [duplicate]

numpy.vectorize takes a function f:a->b and turns it into g:a[]->b[].
This works fine when a and b are scalars, but I can't think of a reason why it wouldn't work with b as an ndarray or list, i.e. f:a->b[] and g:a[]->b[][]
For example:
import numpy as np
def f(x):
return x * np.array([1,1,1,1,1], dtype=np.float32)
g = np.vectorize(f, otypes=[np.ndarray])
a = np.arange(4)
print(g(a))
This yields:
array([[ 0. 0. 0. 0. 0.],
[ 1. 1. 1. 1. 1.],
[ 2. 2. 2. 2. 2.],
[ 3. 3. 3. 3. 3.]], dtype=object)
Ok, so that gives the right values, but the wrong dtype. And even worse:
g(a).shape
yields:
(4,)
So this array is pretty much useless. I know I can convert it doing:
np.array(map(list, a), dtype=np.float32)
to give me what I want:
array([[ 0., 0., 0., 0., 0.],
[ 1., 1., 1., 1., 1.],
[ 2., 2., 2., 2., 2.],
[ 3., 3., 3., 3., 3.]], dtype=float32)
but that is neither efficient nor pythonic. Can any of you guys find a cleaner way to do this?

np.vectorize is just a convenience function. It doesn't actually make code run any faster. If it isn't convenient to use np.vectorize, simply write your own function that works as you wish.
The purpose of np.vectorize is to transform functions which are not numpy-aware (e.g. take floats as input and return floats as output) into functions that can operate on (and return) numpy arrays.
Your function f is already numpy-aware -- it uses a numpy array in its definition and returns a numpy array. So np.vectorize is not a good fit for your use case.
The solution therefore is just to roll your own function f that works the way you desire.

A new parameter signature in 1.12.0 does exactly what you what.
def f(x):
return x * np.array([1,1,1,1,1], dtype=np.float32)
g = np.vectorize(f, signature='()->(n)')
Then g(np.arange(4)).shape will give (4L, 5L).
Here the signature of f is specified. The (n) is the shape of the return value, and the () is the shape of the parameter which is scalar. And the parameters can be arrays too. For more complex signatures, see Generalized Universal Function API.

import numpy as np
def f(x):
return x * np.array([1,1,1,1,1], dtype=np.float32)
g = np.vectorize(f, otypes=[np.ndarray])
a = np.arange(4)
b = g(a)
b = np.array(b.tolist())
print(b)#b.shape = (4,5)
c = np.ones((2,3,4))
d = g(c)
d = np.array(d.tolist())
print(d)#d.shape = (2,3,4,5)
This should fix the problem and it will work regardless of what size your input is. "map" only works for one dimentional inputs. Using ".tolist()" and creating a new ndarray solves the problem more completely and nicely(I believe). Hope this helps.

You want to vectorize the function
import numpy as np
def f(x):
return x * np.array([1,1,1,1,1], dtype=np.float32)
Assuming that you want to get single np.float32 arrays as result, you have to specify this as otype. In your question you specified however otypes=[np.ndarray] which means you want every element to be an np.ndarray. Thus, you correctly get a result of dtype=object.
The correct call would be
np.vectorize(f, signature='()->(n)', otypes=[np.float32])
For such a simple function it is however better to leverage numpy's ufunctions; np.vectorize just loops over it. So in your case just rewrite your function as
def f(x):
return np.multiply.outer(x, np.array([1,1,1,1,1], dtype=np.float32))
This is faster and produces less obscure errors (note however, that the results dtype will depend on x if you pass a complex or quad precision number, so will be the result).

I've written a function, it seems fits to your need.
def amap(func, *args):
'''array version of build-in map
amap(function, sequence[, sequence, ...]) -> array
Examples
--------
>>> amap(lambda x: x**2, 1)
array(1)
>>> amap(lambda x: x**2, [1, 2])
array([1, 4])
>>> amap(lambda x,y: y**2 + x**2, 1, [1, 2])
array([2, 5])
>>> amap(lambda x: (x, x), 1)
array([1, 1])
>>> amap(lambda x,y: [x**2, y**2], [1,2], [3,4])
array([[1, 9], [4, 16]])
'''
args = np.broadcast(None, *args)
res = np.array([func(*arg[1:]) for arg in args])
shape = args.shape + res.shape[1:]
return res.reshape(shape)
Let try
def f(x):
return x * np.array([1,1,1,1,1], dtype=np.float32)
amap(f, np.arange(4))
Outputs
array([[ 0., 0., 0., 0., 0.],
[ 1., 1., 1., 1., 1.],
[ 2., 2., 2., 2., 2.],
[ 3., 3., 3., 3., 3.]], dtype=float32)
You may also wrap it with lambda or partial for convenience
g = lambda x:amap(f, x)
g(np.arange(4))
Note the docstring of vectorize says
The vectorize function is provided primarily for convenience, not for
performance. The implementation is essentially a for loop.
Thus we would expect the amap here have similar performance as vectorize. I didn't check it, Any performance test are welcome.
If the performance is really important, you should consider something else, e.g. direct array calculation with reshape and broadcast to avoid loop in pure python (both vectorize and amap are the later case).

The best way to solve this would be to use a 2-D NumPy array (in this case a column array) as an input to the original function, which will then generate a 2-D output with the results I believe you were expecting.
Here is what it might look like in code:
import numpy as np
def f(x):
return x*np.array([1, 1, 1, 1, 1], dtype=np.float32)
a = np.arange(4).reshape((4, 1))
b = f(a)
# b is a 2-D array with shape (4, 5)
print(b)
This is a much simpler and less error prone way to complete the operation. Rather than trying to transform the function with numpy.vectorize, this method relies on NumPy's natural ability to broadcast arrays. The trick is to make sure that at least one dimension has an equal length between the arrays.

Vectorised assignment to tensor

I'd like to assign multiple values to a tensor, but it seems that it's not supported at least in the way that is possible using numpy.
a = np.zeros((4, 4))
v = np.array([0, 2, 3, 1])
r = np.arange(4)
a[r, v] = 1
>>> a
array([[1., 0., 0., 0.],
[0., 0., 1., 0.],
[0., 0., 0., 1.],
[0., 1., 0., 0.]])
The above works, but the tensorflow equivalent doesn't:
import tensorflow as tf
a = tf.zeros((4, 4))
v = tf.Variable([0, 2, 3, 1])
r = tf.range(4)
a[r, v].assign(1)
TypeError: Only integers, slices, ellipsis, tf.newaxis and scalar tensors are valid indices, got <tf.Tensor: shape=(4,), dtype=int32, numpy=array([0, 1, 2, 3])>
How could this be achieved? Are loops the only option? In my case the resulting array is indeed only slices of an identity matrix rearranged, so maybe that could be taken advantage of somehow.

Your example, which is updating a zero tensor at some indices to a certain value is most of time achieved through tf.scatter_nd :
idx = tf.stack([r,v],axis=-1)
tf.scatter_nd(idx, updates=tf.ones(4), shape=(4,4))
For more complex cases, you can look at the following functions:
tf.tensor_scatter_nd_add: Adds sparse updates to an existing tensor according to indices.
tf.tensor_scatter_nd_sub: Subtracts sparse updates from an existing tensor according to indices.
tf.tensor_scatter_nd_max: to copy element-wise maximum values from one tensor to another.
tf.tensor_scatter_nd_min: to copy element-wise minimum values from one tensor to another.
tf.tensor_scatter_nd_update: Scatter updates into an existing tensor according to indices.
You can read more in the guide: Introduction to tensor slicing

Assign arr with numpy fancy indexing

I really hope this is not a duplicate and this is probably a very stupid question. Sorry ;)
Problem:
I have a greyscale image with values/classes 1 and 2 and I want to convert/map this to a color image where 1 equals yellow and 2 equals blue.
import numpy as np
import cv2
result=cv2.imread("image.png", cv2.IMREAD_GRAYSCALE)
result[result==2]=[15,100,100]
result[result==1]=[130,255,255]
But this is failing with the error ValueError: NumPy boolean array indexing assignment cannot assign 3 input values to the 1995594 output values where the mask is true
I think I very close to the solution, but I don't get it.
Thanks in Advance for your help!

result is a Numpy array and is typed, its type being an integer and you try to assign to an integer slot a triple of integers… no good.
What you want to do is creating an empty color image, with the same dimensions of result, and assigning to the last axis the requested triples.
I have not installed cv2 but you can look at the following code to have an idea on how to proceed.
Equivalent to what you have done, the same error
In [36]: import numpy as np
In [37]: a = np.random.randint(0,2,(2,4))
In [38]: a
Out[38]:
array([[1, 0, 0, 0],
[0, 1, 0, 1]])
In [39]: a[a==1] = (1,2,3)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-39-24af4c8dbf5a> in <module>
----> 1 a[a==1] = (1,1)
ValueError: NumPy boolean array indexing assignment cannot assign 2 input values to the 3 output values where the mask is true
Now, allocate a 3D array and apply indexing to it, assigning by default to the last axis
In [40]: b = np.zeros((2,4,3))
In [41]: b[a==1] = (1,2,3)
In [42]: b
Out[42]:
array([[[1., 2., 3.],
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]],
[[0., 0., 0.],
[1., 2., 3.],
[0., 0., 0.],
[1., 2., 3.]]])
We have two inner matrices (corresponding to the two rows of a), in each matrix four rows (corresponding to the four columns of a) and finally the columns are the RGB triples that you need.
I don't know exactly how the data is arranged in a cv2 image but I think you have to do minor adjustements, if any at all.

Thanks to #gboffi i got an answer. I guess I was hoping that there is a more pythonic way of doing this, but ok.
# Load in Image File
img=cv2.imread("imgfile", cv2.IMREAD_GRAYSCALE)
# Create new array with the shape of the original image
color=np.zeros((img.shape[0],result.shape[1],3))
# Note that opencv needs BGR and values from 0..1
color[result==1]=(0.84,117/225,2/225)
color[result==2]=(78/225,173/225,240/225)

How to multiply diagonal elements by each other using numpy?

For the purpose of this exercise, let's consider a matrix where the element m_{i, j} is given by the rule m_{i, j} = i*j if i == j and 0 else.
Is there an easy "numpy" way of calculating such a matrix without having to resort to if statements checking for the indices?

You can use the numpy function diag to construct a diagonal matrix if you give it the intended diagonal as a 1D array as input.
So you just need to create that, like [i**2 for i in range (N)] with N the dimension of the matrix.

You could use the identity matrix given by numpy.identity(n) and then multiply it by a n dimensional vector.

Assuming you have a squared matrix, you can do this:
import numpy as np
ary = np.zeros((4, 4))
_ = [ary.__setitem__((i, i), i**2) for i in range(ary.shape[0])]
print(ary)
# array([[0., 0., 0., 0.],
# [0., 1., 0., 0.],
# [0., 0., 4., 0.],
# [0., 0., 0., 9.]])

Create a numpy array according to another array along with indices array

I have a numpy array(eg., a = np.array([ 8., 2.])), and another array which stores the indices I would like to get from the former array. (eg., b = np.array([ 0., 1., 1., 0., 0.]).
What I would like to do is to create another array from these 2 arrays, in this case, it should be: array([ 8., 2., 2., 8., 8.])
of course, I can always use a for loop to achieve this goal:
for i in range(5):
c[i] = a[b[i]]
I wonder if there is a more elegant method to create this array. Something like c = a[b[0:5]] (well, this apparently doesn't work)

Only integer arrays can be used for indexing, and you've created b as a float64 array. You can get what you're looking for if you explicitly convert to integer:
bi = np.array(b, dtype=int)
c = a[bi[0:5]]