My program creates a numpy array within a for loop. For example it creates array with shape (100*30*10), then (160*30*10) and then may be (120*30*10) . I have to append the above to an empty numpy array such that , at the end of the loop, it will be a numpy array with shape (380*30*10) (i.e sum of 100+160+120) . The second and third dimension doesnt change in the numpy array.
How can I do the above in python. I tried the following.
np_model = np.append(np_model,np_temp1)
print("Appended model shape is",np_model.shape)
np_label = np.append(np_label,np_temp2)
print("Appended label shape is",np_label.shape)
The np_model is an empty array which I have defined as np_model = np.empty(1,30,10) and np_label as np_label = np.empty(1 ,str)
np_temp1 corresponds to array within each for loop like 100*30*10,120*30*10 etc and np_temp2 is a string with "item1","item2" etc
The np_label is a string numpy array with 1 label corresponding to np_temp1.shape[0]. But the result I get in np_model is flattened array with size 380*30*10 = 1140000
Any help is appreciated.
you can use numpy concatenate function, append the output numpy(s) to a list and then feed it to the concatenate function:
empty_list = []
x = np.zeros([10, 20, 4])
y = np.zeros([12, 20, 4])
empty_list.append(x)
empty_list.append(y)
z = np.concatenate(empty_list, axis=0)
print(x.shape, y.shape, z.shape)
(10, 20, 4) (12, 20, 4) (22, 20, 4)
As #Nullman suggested in comment(np.vstack)
You can create empty array like this >>> np_model = np.empty((0,30,10))
>>> np_model = np.empty((0,30,10))
>>> a = np.random.rand(100,30,10)
>>> b = np.random.rand(160,30,10)
>>> c = np.random.rand(120,30,10)
# It can done by one-line like`np_model = np.vstack((a,b,c))`
# but i guess you have loop dependency here
>>> np_model = np.vstack((np_model,a))
>>> np_model = np.vstack((np_model,b))
>>> np_model = np.vstack((np_model,c))
>>> np_model.shape
(380, 30, 10)
To specifically answer your question towards starting with an empty array, that'd be my solution, solely using np.concatenate:
import numpy as np
# Some arrays to append in a loop
arrays = (
np.random.rand(100, 30, 10),
np.random.rand(160, 30, 10),
np.random.rand(120, 30, 10)
)
# Initial empty array
array = np.zeros((0, 30, 10))
# Appending arrays in loop
for a in arrays:
array = np.concatenate((array, a), axis=0)
# Output shape
print(array.shape)
Output:
(380, 30, 10)
Hope that helps!
----------------------------------------
System information
----------------------------------------
Platform: Windows-10-10.0.16299-SP0
Python: 3.8.1
NumPy: 1.18.1
----------------------------------------
Related
How do you reshape a (55, 11) numpy array to a (55, 11, 1) numpy array?
Attempts:
Simply doing numpy_array.reshape(-1, 1) without any loop produces a flat array that is not 3D.
The following for loop produces a "cannot
broadcast error":
for i in range(len(numpy_array)):
numpy_array[i] = numpy_array[i].reshape(-1, 1)
Maybe you are looking for numpy.expand_dims(https://numpy.org/doc/stable/reference/generated/numpy.expand_dims.html)?
import numpy
a = numpy.random.rand(55,11)
print(a.shape) # 55,11
print(numpy.expand_dims(a, 2).shape) # 55, 11, 1
Add a newaxis to the array
my_array = np.arange(55*11).reshape(55,11)
my_array.shape
# (55, 11)
# add new axis
new_array = my_array[...,None]
new_array.shape
# (55, 11, 1)
Can specify new shape in reshape too:
new_array = my_array.reshape(*my_array.shape, 1)
new_array.shape
# (55, 11, 1)
One of the answers recommends using expand_dims. That's a good answer, but if you look at its code, and strip off some generalities, all it is doing is:
In [409]: a = np.ones((2,3)); axis=(2,)
...: out_ndim = 2+1
...: shape_it = iter(a.shape)
...: shape = [1 if ax in axis else next(shape_it) for ax in range(out_ndim)]
In [410]: shape
Out[410]: [2, 3, 1]
followed by a return a.reshape(shape).
In other words, the function call is just hiding the obvious, expand a (x,y) to (x,y,1) with
a.reshape(x,y,1)
Are you seeking some 3d 'magic' akin to the -1 in numpy_array.reshape(-1, 1)?
Personally I like to use None to add dimensions, so prefer the other answer [...,None]. But functionally it's all the same.
lets assume we have a tensor representing an image of the shape (910, 270, 1) which assigned a number (some index) to each pixel with width=910 and height=270.
We also have a numpy array of size (N, 3) which maps a 3-tuple to an index.
I now want to create a new numpy array of shape (920, 270, 3) which has a 3-tuple based on the original tensor index and the mapping-3-tuple-numpy array. How do I do this assignment without for loops and other consuming iterations?
This would look simething like:
color_image = np.zeros((self._w, self._h, 3), dtype=np.int32)
self._colors = np.array(N,3) # this is already present
indexed_image = torch.tensor(920,270,1) # this is already present
#how do I assign it to this numpy array?
color_image[indexed_image.w, indexed_image.h] = self._colors[indexed_image.flatten()]
Assuming you have _colors, and indexed_image. Something that ressembles to:
>>> indexed_image = torch.randint(0, 10, (920, 270, 1))
>>> _colors = np.random.randint(0, 255, (N, 3))
A common way of converting a dense map to a RGB map is to loop over the label set:
>>> _colors = torch.FloatTensor(_colors)
>>> rgb = torch.zeros(indexed_image.shape[:-1] + (3,))
>>> for lbl in range(N):
... rgb[lbl == indexed_image[...,0]] = _colors[lbl]
I have an empty list: x = [].
I have a numpy array, y, of shape: (180, 161). I can't necessarily define x to be an np.empty of a particular shape, because I won't know the shape of y ahead of time.
I want to append y to x so that x will have a .shape of (1, 180, 161).
Then if I append more, I want it to be (n, 180, 161)
I tried .append and .stack, but I've had a variety of errors:
TypeError: only size-1 arrays can be converted to Python scalars
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 3 dimension(s) and the array at index 1 has 2 dimension(s)
And so on. It seems that this should be simple, but it's strangely difficult.
Assuming all items in x have the same shape, you can first construct a list and then construct the NumPy array from the list.
There, you have two options:
np.array() which is faster but not flexible
np.stack() which is slower but allows you to choose over which axis should the stack happen (it is roughly equivalent to np.array().transpose(...).copy()
The code would look like:
import numpy as np
n = 100
x = [np.random.randint(0, 10, (10, 20)) for _ in range(n)]
# same as: y = np.stack(x, 0)
y = np.array(x)
print(y.shape)
# (100, 10, 20)
Of course this line:
x = [np.random.randint(0, 10, (10, 20)) for _ in range(n)]
can be replaced with:
x = []
for _ in range(n):
x.append(np.random.randint(0, 10, (10, 20)))
You could also use np.append(), e.g.:
def stacker(arrs):
result = arrs[0][None, ...]
for arr in arrs[1:]:
result = np.append(result, arr[None, ...], 0)
return result
but with horrific performances:
n = 1000
shape = (100, 100)
x = [np.random.randint(0, n, shape) for _ in range(n)]
%timeit np.array(x)
# 10 loops, best of 3: 21.1 ms per loop
%timeit np.stack(x)
# 10 loops, best of 3: 21.6 ms per loop
%timeit stacker(x)
# 1 loop, best of 3: 11 s per loop
and, as you can see, performance-wise, the list-based method is way faster.
You can reshape y to be (1, *y.shape).
Then for appending an array you can say:
y_1 = np.vstack((y, new_arr))
where y_1.shape produces a (2, *y.shape) numpy array.
To save memory you can say y = np.vstack((y, new_arr))
You might have to reshape your array to (1, *y.shape) however.
This is a very basic example:
import numpy as np
a = np.ones((1,2,3))
b = np.ones((1,2,3))
np.vstack((a,b)).shape # (2,2,3)
Let me know if this helps!
If you keep x as a list then if you just want to maintain the shape by appending, it is possible:
>>> import numpy as np
>>> x = []
>>> y = np.arange(12).reshape(3,4)
>>> x.append(y)
>>> np.shape(x)
(1, 3, 4)
>>> x.append(y)
>>> np.shape(x)
(2, 3, 4)
>>> for i in range(10):
... x.append(y)
>>> np.shape(x)
(12, 3, 4)
But considering you are dealing with np.arrays it may not be convenient for you to keep x as list, so you may try this:
>>> x = np.array(x)
>>> x.shape
(12, 3, 4)
>>> y[None,...].shape
(1, 3, 4)
>>> np.append(x, y[None,...],axis=0).shape
(13, 3, 4)
Word of caution:
As pointed out by #hpaulj :
np.append should be avoided, as it is extremely slow, probably only faster than:
x = np.array([*x, y])
The correct usage would be:
x = np.concatenate([x, y[None,...]], axis=0)
Either way, concatenating or appending is generally a speed bump in numpy. So unless you absolutely need to create an array this way, you should work with lists. Also most functions applied to np.arrays work on lists as well. Note, functions applied to arrays, not methods of an np.array object. For example:
>>> x = list((1, 2, 3, 4))
>>> np.shape(x)
(4,)
>>> x.shape
Traceback (most recent call last):
File "<ipython-input-100-9f2b259887ef>", line 1, in <module>
x.shape
AttributeError: 'list' object has no attribute 'shape'
So I would suggest appending to list, and then after you have done appending all the arrays, convert the list to np.array if you require.
Suppose I have a 5x10x3 array, which I interpret as 5 'sub-arrays', each consisting of 10 rows and 3 columns. I also have a seperate 1D array of length 5, which I call b.
I am trying to insert a new column into each sub-array, where the column inserted into the ith (i=0,1,2,3,4) sub-array is a 10x1 vector where each element is equal to b[i].
For example:
import numpy as np
np.random.seed(777)
A = np.random.rand(5,10,3)
b = np.array([2,4,6,8,10])
A[0] should look like:
A[1] should look like:
And similarly for the other 'sub-arrays'.
(Notice b[0]=2 and b[1]=4)
What about this?
# Make an array B with the same dimensions than A
B = np.tile(b, (1, 10, 1)).transpose(2, 1, 0) # shape: (5, 10, 1)
# Concatenate both
np.concatenate([A, B], axis=-1) # shape: (5, 10, 4)
One method would be np.pad:
np.pad(A, ((0,0),(0,0),(0,1)), 'constant', constant_values=[[[],[]],[[],[]],[[],b[:, None,None]]])
# array([[[9.36513084e-01, 5.33199169e-01, 1.66763960e-02, 2.00000000e+00],
# [9.79060284e-02, 2.17614285e-02, 4.72452812e-01, 2.00000000e+00],
# etc.
Or (more typing but probably faster):
i,j,k = A.shape
res = np.empty((i,j,k+1), np.result_type(A, b))
res[...,:-1] = A
res[...,-1] = b[:, None]
Or dstack after broadcast_to:
np.dstack([A,np.broadcast_to(b[:,None],A.shape[:2])]
I have a list with numpy.ndarrays - each of shape (33,1,8,45,3)
Problem that when i concatenate the list using a = np.concatenate(list)
The output shape of a becomes
print a.shape
(726,1,8,45,3)
instead of shape (22,33,1,8,45,3).
How do I cleanly concatenate the list, without having to change the input.
You can use numpy.array() or numpy.stack():
import numpy
a = [numpy.random.rand(33,1,8,45,3) for i in range(22)]
b = numpy.array(a)
b.shape # (22, 33, 1, 8, 45, 3)
c = numpy.stack(a, axis=0)
c.shape # (22, 33, 1, 8, 45, 3)
np.concatenate:
Join a sequence of arrays along an existing axis.
np.stack:
Stack a sequence of arrays along a new axis.
a = np.ones((3, 4))
b = np.stack([a, a])
print(b.shape) # (2, 3, 4)