I have a numpy array named PixelData of unknown shape, and I am using the following condition to filter values in the array greater than some value x using a mask:
PixelData[PixelData>=x] = PixelData[PixelData>=x] - x
When I convert this numpy array to a tensor, I cannot perform the same masking operation. I have tried using tf.where as follows:
PixelData = tf.where(PixelData>=x, PixelData - x, PixelData)
In the official documentation, they always seem to define the mask dimensions in advance to equal the dimensions of the tensor being masked, but then they talk about the dimensions being broadcasted automatically, so I am a bit confused. Are these two functions equivalent? Are there any situations where they may produce different outputs?
Not sure what PixelData looks like, but here is working example with both methods:
import numpy as np
import tensorflow as tf
x = 2
np_pixel_data = np.array([[3, 4, 5, 1],
[6, 4, 2, 5]], dtype=np.float32)
np_pixel_data[np_pixel_data>=x] = np_pixel_data[np_pixel_data>=x] - x
tf_pixel_data = tf.constant([[3, 4, 5, 1],
[6, 4, 2, 5]], dtype=tf.float32)
tf_pixel_data = tf.where(tf.greater_equal(tf_pixel_data, x), tf_pixel_data - x, tf_pixel_data)
print(np_pixel_data)
print(tf_pixel_data)
[[1. 2. 3. 1.]
[4. 2. 0. 3.]]
tf.Tensor(
[[1. 2. 3. 1.]
[4. 2. 0. 3.]], shape=(2, 4), dtype=float32)
You might have some minor rounding differences, but nothing significant.
Related
This relates to this earlier post: Numpy random choice of tuples
I have a 2D numpy array and want to choose from it using a 2D probability array. The only way I could think to do this was to flatten and then use the modulo and remainder to convert the result back to a 2D index
import numpy as np
# dummy data
x=np.arange(100).reshape(10,10)
# dummy probability array
p=np.zeros([10,10])
p[4:7,1:4]=1.0/9
xy=np.random.choice(x.flatten(),1,p=p.flatten())
index=[int(xy/10),(xy%10)[0]] # convert back to index
print(index)
which gives
[5, 2]
but is there a cleaner way that avoids flattening and the modulo? i.e. I could pass a list of coordinate tuples as x, but how can I then handle the weights?
I don't think it's possible to directly specify a 2D shaped array of probabilities. So raveling should be fine. However to get the corresponding 2D shaped indices from the flat index you can use np.unravel_index
index= np.unravel_index(xy.item(), x.shape)
# (4, 2)
For multiple indices, you can just stack the result:
xy=np.random.choice(x.flatten(),3,p=p.flatten())
indices = np.unravel_index(xy, x.shape)
# (array([4, 4, 5], dtype=int64), array([1, 2, 3], dtype=int64))
np.c_[indices]
array([[4, 1],
[4, 2],
[5, 3]], dtype=int64)
where np.c_ stacks along the right hand axis and gives the same result as
np.column_stack(indices)
You could use numpy.random.randint to generate an index, for example:
# assumes p is a square array
ij = np.random.randint(p.shape[0], size=p.ndim) # size p.ndim = 2 generates 2 coords
# need to convert to tuple to index correctly
p[tuple(i for i in ij))]
>>> 0.0
You can also index multiple random values at once:
ij = np.random.randint(p.shape[0], size=(p.ndim, 5)) # get 5 values
p[tuple(i for i in ij))]
>>> array([0. , 0. , 0. , 0.11111111, 0. ])
I have read most related questions here, but I cannot seem to figure out how to use np.pad in this case. Maybe it is not meant for this particular problem.
Let's say I have a list of Numpy arrays. Every array is the same length, e.g. 2. The list itself has to be padded to be e.g. 5 arrays and can be transformed into a numpy array as well. The padded elements should be arrays filled with zeroes. As an example
arr = [array([0, 1]), array([1, 0]), array([1, 1])]
expected_output = array([array([0, 1]), array([1, 0]), array([1, 1]), array([0, 0]), array([0, 0])])
The following seems to work, but I feel there must be a better and more efficient way. In reality this is run hundreds of thousands if not millions of times so speed is important. Perhaps with np.pad?
import numpy as np
def pad_array(l, item_size, pad_size=5):
s = len(l)
if s < pad_size:
zeros = np.zeros(item_size)
for _ in range(pad_size-s):
# not sure if I need a `copy` of zeros here?
l.append(zeros)
return np.array(l)
B = [np.array([0,1]), np.array([1,0]), np.array([1,1])]
AB = pad_array(B, 2)
print(AB)
It seems like you want to pad zeros at the end of the axis 0, speaking in numpy terms. So what you need is,
output = numpy.pad(arr, ((0,2),(0,0)), 'constant')
The trick is the pad_width parameter, which you need to specify as pad_width=((0,2),(0,0)) to get your expected output. This is you telling pad() to insert 0 padding at the beginning and 2 padding at the end of the axis 0, and to insert 0 padding at the beginning and 0 padding at the end of the axis 1. The format of pad_width is ((before_1, after_1), … (before_N, after_N)) according to the documentation
mode='constant' tells pad() to pad with the value specified by parameter constant_values which defaults to 0.
You could re-write your function like this:
import numpy as np
def pad_array(l, item_size, pad_size=5):
if pad_size < len(l):
return np.array(l)
s = len(l)
res = np.zeros((pad_size, item_size)) # create an array of (item_size, pad_size)
res[:s] = l # set the first rows equal to the elements of l
return res
B = [np.array([0, 1]), np.array([1, 0]), np.array([1, 1])]
AB = pad_array(B, 2)
print(AB)
Output
[[0. 1.]
[1. 0.]
[1. 1.]
[0. 0.]
[0. 0.]]
The idea is to create an array of zeroes and then fill the first rows with the values from the input list.
If X = [[ 1. 1. 1. 1. 1. 1.]] and Y = [[ 0. 0. 0. 0.]] - how can I concatenate the two vectors to form a single vector along column?
I did the following but it didn't work:
import tensorflow as tf
X = tf.constant(1.0, shape=[1, 6])
Y = tf.zeros(shape=[1,4])
XY = tf.concat((X,Y), axis = 0)
sess = tf.Session()
print(sess.run(XY))
If you want to concat them on axis 0, then their size must be equal
Assuming that you don't want it,
you need to set axis = 1 in the tf.concat method
While generating a linspace array in Numpy we get an array of the form (len(array), ), i.e. it doesn't have any 2nd dimension. How do I generate a similar array and initialize it using Numpy zeros? Because it takes a 2nd argument, like 1, so I get (len(array), 1) while initializing, which I wanted to avoid if possible.
Eg. np.linspace(0,10,5) = [0, 2.5, 5, 7.5, 10] ;
It's array dimension is (5, ).
On the other hand, a zeros array is defined as np.zeros((5,1)) and our output is a vector [0 0 0 0 0] ^ (Transpose). I wanted to be a flat array not like a vector.
Is there a way?
your first argument (5,1) is defining the shape of the array as a 5x1 explicitly 2d shape. Just pass (5,), or more explicitly as follows:
import numpy as np
z = np.zeros(shape=(5,), dtype=float)
print(z)
print(z.shape)
output is:
[ 0. 0. 0. 0. 0.]
(5,)
I searched stackoverflow but could not find an answer to this specific question. Sorry if it is a naive question, I am a newbie to python.
I have several 2d arrays (or lists) that I would like to read into a 3d array (list) in python. In Matlab, I can simply do
for i=1:N
# read 2d array "a"
newarray(:,:,i)=a(:,:)
end
so newarray is a 3d array with "a" being the 2d slices arranged along the 3rd dimension.
Is there a simple way to do this in python?
Edit: I am currently trying the following:
for file in files:
img=mpimg.imread(file)
newarray=np.array(0.289*cropimg[:,:,0]+0.5870*cropimg[:,:,1]+0.1140*cropimg[:,:,2])
i=i+1
I tried newarray[:,:,i] and it gives me an error
NameError: name 'newarray' is not defined
Seems like I have to define newarray as a numpy array? Not sure.
Thanks!
If you're familiar with MATLAB, translating that into using NumPy is fairly straightforward.
Lets say you have a couple arrays
a = np.eye(3)
b = np.arange(9).reshape((3, 3))
print(a)
# [[ 1. 0. 0.]
# [ 0. 1. 0.]
# [ 0. 0. 1.]]
print(b)
# [[0 1 2]
# [3 4 5]
# [6 7 8]]
If you simply want to put them into another dimension, pass them both to the array constructor in an iterable (e.g. a list) like so:
x = np.array([a, b])
print(x)
# [[[ 1. 0. 0.]
# [ 0. 1. 0.]
# [ 0. 0. 1.]]
#
# [[ 0. 1. 2.]
# [ 3. 4. 5.]
# [ 6. 7. 8.]]]
Numpy is smart enough to recognize the arrays are all the same size and creates a new dimension to hold it all.
print(x.shape)
# (2, 3, 3)
You can loop through it, but if you want to apply the same operations to it across some dimensions, I would strongly suggest you use broadcasting so that NumPy can vectorize the operation and it runs a whole lot faster.
For example, across one dimension, lets multiply one slice by 2, another by 3. (If it's not a pure scalar, we need to reshape the array to the same number of dimensions to broadcast, then the size on each needs to either match the array or be 1). Note that I'm working along the 0th axis, your image is probably different. I don't have a handy image to load up to toy with
y = x * np.array([2, 3]).reshape((2, 1, 1))
print(y)
#[[[ 2. 0. 0.]
# [ 0. 2. 0.]
# [ 0. 0. 2.]]
#
# [[ 0. 3. 6.]
# [ 9. 12. 15.]
# [ 18. 21. 24.]]]
Then we can add them up
z = np.sum(y, axis=0)
print(z)
#[[ 2. 3. 6.]
# [ 9. 14. 15.]
# [ 18. 21. 26.]]
If you're using NumPy arrays, you can translate almost directly from Matlab:
for i in range(1, N+1):
# read 2d array "a"
newarray[:, :, i] = a[:, :]
Of course you'd probably want to use range(N), because arrays use 0-based indexing. And obviously you're going to need to pre-create newarray in some way, just as you'd have to in Matlab, but you can translate that pretty directly too. (Look up the zeros function if you're not sure how.)
If you're using lists, you can't do this directly—but you probably don't want to anyway. A better solution would be to build up a list of 2D lists on the fly:
newarray = []
for i in range(N):
# read 2d list of lists "a"
newarray.append(a)
Or, more simply:
newarray = [read_next_2d_list_of_lists() for i in range(N)]
Or, even better, make that read function a generator, then just:
newarray = list(read_next_2d_list_of_lists())
If you want to transpose the order of the axes, you can use the zip function for that.