Related
Consider 2D Numpy array A and in-place function x like
A = np.arange(9).reshape(3,3)
def x(M):
M[:,2] = 0
Now, I have a list (or 1D numpy array) L pointing the rows, I want to select and apply the function f on them like
L = [0, 1]
x(A[L, :])
where the output will be written to A. Since I used index access to A, the matrix A is not affected at all:
A = array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
What I actually need is to slice the matrix such as
x(A[:2, :])
giving me the desired output
A = array([[0, 1, 0],
[3, 4, 0],
[6, 7, 8]])
The question is now, how to provide Numpy array slicing by the list L (either any automatic conversion of list to slice or if there is any build in function for that), because I am not able to convert the list L easily to slice like :2 in this case.
Note that I have both large matrix A and list L in my problem - that is the reason, why I would need the in-place operations to control the available memory.
Can you modify the function so as you can pass slice L inside it:
def func(M,L):
M[L,2] = 0
func(A,L)
print(A)
Out:
array([[0, 1, 0],
[3, 4, 0],
[6, 7, 8]])
I wish to vstack a numpy.array (like building a list) but, I cannot initialize the numpy.array with the correct shape to use numpy.append(numpy.empty/zero/like_empty, etc. did not do the trick... anyway. Finally, I figure the two pieces of code below. Is there someyhing more pythonic? I am using python 3.6.9
import numpy as np
a=[]
n=4
for i in range(n):
'''
some calculation resultinng for example in an numpy.array([[i,i+1,i+2])
'''
a.append(np.array([i,i+1,i+2]))
a=np.array(a).reshape(3,n)
print(a)
or because I prefer to mantain it as a numpy array inside the loop:
import numpy as np
a=np.array([])
n=4
for i in range(n):
'''
some calculation resultinng for example in an numpy.array [i,i+1,i+2]
'''
if a.size == 0:
a=np.array([i,i+1,i+2])
else:
a=np.vstack((a,np.array([i,i+1,i+2])))
print(a)
both output:
[[0 1 2]
[1 2 3]
[2 3 4]
[3 4 5]]
Your first use, with list append:
In [146]: alist=[]
In [147]: for i in range(4):
...: alist.append(np.arange(i,i+3))
...:
In [148]: alist
Out[148]: [array([0, 1, 2]), array([1, 2, 3]), array([2, 3, 4]), array([3, 4, 5])]
and make the array:
In [149]: np.array(alist)
Out[149]:
array([[0, 1, 2],
[1, 2, 3],
[2, 3, 4],
[3, 4, 5]])
or since vstack is happy with a list of arrays:
In [150]: np.vstack(alist)
Out[150]:
array([[0, 1, 2],
[1, 2, 3],
[2, 3, 4],
[3, 4, 5]])
You could use vstack in the loop:
In [151]: arr = np.zeros((0,3),int)
In [152]: for i in range(4):
...: arr = np.vstack((arr, np.arange(i,i+3)))
...:
In [153]: arr
Out[153]:
array([[0, 1, 2],
[1, 2, 3],
[2, 3, 4],
[3, 4, 5]])
This has two problems:
it is slower; list append operates in-place simply adding a pointer to the list. vstack makes whole new array each time, with a full copy!
it is harder to initialize, as you found out. You actually have to understand array shapes, and what concatenate does when it combines 2 or more arrays. Here I started with a (0,3) array.
np.array([np.arange(i, i+3) for i in range(n)])
I want to create a numpy array within a numpy array. If i do it with normal python its something like
a = [[1,2], [3,4]]
a[0][1] = [1,1,1]
print a
The result is [[1, [1, 1, 1]], [3, 4]]
How can I achieve the same using numpy arrays? The code I have is:
a = np.array([(1, 2, 3),(4, 5, 6)])
b = np.array([1,1,1])
a[0][1] = b
a as created is dtype int. Each element can only be another integer:
In [758]: a = np.array([(1, 2, 3),(4, 5, 6)])
...: b = np.array([1,1,1])
...:
In [759]: a
Out[759]:
array([[1, 2, 3],
[4, 5, 6]])
In [760]: b
Out[760]: array([1, 1, 1])
In [761]: a[0,1]=b
...
ValueError: setting an array element with a sequence.
You can make another dtype of array, one that holds pointers to objects, much as list does:
In [762]: aO = a.astype(object)
In [763]: aO
Out[763]:
array([[1, 2, 3],
[4, 5, 6]], dtype=object)
Now it is possible to replace one of those element pointers with a pointer to b array:
In [765]: aO[0,1]=b
In [766]: aO
Out[766]:
array([[1, array([1, 1, 1]), 3],
[4, 5, 6]], dtype=object)
But as asked in the comments - why do you want/need to do this? What are you going to do with such an array? It is possible to do some numpy math on such an array, but as shown in some recent SO questions, it is hit-or-miss. It is also slower.
As far as I know, you cannot do this. Numpy arrays cannot have entries of varying shape. Your request to make an array like [[1, [1, 1, 1]], [3, 4]] is impossible. However, you could make a numpy matrix of dimensions (3x2x3) to get
[
[
[1,0,0],
[1,1,1],
[0,0,0],
]
[
[3,0,0],
[4,0,0],
[0,0,0]
]
]
Your only option is to pad empty elements with some number (I used 0s above) or use another data structure.
I tried the following:
>>> a = np.array([1,2,3])
>>> b = np.array([4,5,6])
>>> np.concatenate((a,b), axis=0)
array([1, 2, 3, 4, 5, 6])
>>> np.concatenate((a,b), axis=1)
array([1, 2, 3, 4, 5, 6])
However, I'd expect at least that one result looks like this
array([[1, 2, 3],
[4, 5, 6]])
Why is it not concatenated vertically?
Because both a and b have only one axis, as their shape is (3), and the axis parameter specifically refers to the axis of the elements to concatenate.
this example should clarify what concatenate is doing with axis. Take two vectors with two axis, with shape (2,3):
a = np.array([[1,5,9], [2,6,10]])
b = np.array([[3,7,11], [4,8,12]])
concatenates along the 1st axis (rows of the 1st, then rows of the 2nd):
np.concatenate((a,b), axis=0)
array([[ 1, 5, 9],
[ 2, 6, 10],
[ 3, 7, 11],
[ 4, 8, 12]])
concatenates along the 2nd axis (columns of the 1st, then columns of the 2nd):
np.concatenate((a, b), axis=1)
array([[ 1, 5, 9, 3, 7, 11],
[ 2, 6, 10, 4, 8, 12]])
to obtain the output you presented, you can use vstack
a = np.array([1,2,3])
b = np.array([4,5,6])
np.vstack((a, b))
array([[1, 2, 3],
[4, 5, 6]])
You can still do it with concatenate, but you need to reshape them first:
np.concatenate((a.reshape(1,3), b.reshape(1,3)))
array([[1, 2, 3],
[4, 5, 6]])
Finally, as proposed in the comments, one way to reshape them is to use newaxis:
np.concatenate((a[np.newaxis,:], b[np.newaxis,:]))
If the actual problem at hand is to concatenate two 1-D arrays vertically, and we are not fixated on using concatenate to perform this operation, I would suggest the use of np.column_stack:
In []: a = np.array([1,2,3])
In []: b = np.array([4,5,6])
In []: np.column_stack((a, b))
array([[1, 4],
[2, 5],
[3, 6]])
A not well known feature of numpy is to use r_. This is a simple way to build up arrays quickly:
import numpy as np
a = np.array([1,2,3])
b = np.array([4,5,6])
c = np.r_[a[None,:],b[None,:]]
print(c)
#[[1 2 3]
# [4 5 6]]
The purpose of a[None,:] is to add an axis to array a.
a = np.array([1,2,3])
b = np.array([4,5,6])
np.array((a,b))
works just as well as
np.array([[1,2,3], [4,5,6]])
Regardless of whether it is a list of lists or a list of 1d arrays, np.array tries to create a 2d array.
But it's also a good idea to understand how np.concatenate and its family of stack functions work. In this context concatenate needs a list of 2d arrays (or any anything that np.array will turn into a 2d array) as inputs.
np.vstack first loops though the inputs making sure they are at least 2d, then does concatenate. Functionally it's the same as expanding the dimensions of the arrays yourself.
np.stack is a new function that joins the arrays on a new dimension. Default behaves just like np.array.
Look at the code for these functions. If written in Python you can learn quite a bit. For vstack:
return _nx.concatenate([atleast_2d(_m) for _m in tup], 0)
Suppose you have 3 NumPy arrays (A, B, C). You can contact these arrays vertically like this:
import numpy as np
np.concatenate((A, B, C), axis=1)
np.shape
How does one add rows to a numpy array?
I have an array A:
A = array([[0, 1, 2], [0, 2, 0]])
I wish to add rows to this array from another array X if the first element of each row in X meets a specific condition.
Numpy arrays do not have a method 'append' like that of lists, or so it seems.
If A and X were lists I would merely do:
for i in X:
if i[0] < 3:
A.append(i)
Is there a numpythonic way to do the equivalent?
Thanks,
S ;-)
You can do this:
newrow = [1, 2, 3]
A = numpy.vstack([A, newrow])
What is X? If it is a 2D-array, how can you then compare its row to a number: i < 3?
EDIT after OP's comment:
A = array([[0, 1, 2], [0, 2, 0]])
X = array([[0, 1, 2], [1, 2, 0], [2, 1, 2], [3, 2, 0]])
add to A all rows from X where the first element < 3:
import numpy as np
A = np.vstack((A, X[X[:,0] < 3]))
# returns:
array([[0, 1, 2],
[0, 2, 0],
[0, 1, 2],
[1, 2, 0],
[2, 1, 2]])
As this question is been 7 years before, in the latest version which I am using is numpy version 1.13, and python3, I am doing the same thing with adding a row to a matrix, remember to put a double bracket to the second argument, otherwise, it will raise dimension error.
In here I am adding on matrix A
1 2 3
4 5 6
with a row
7 8 9
same usage in np.r_
A = [[1, 2, 3], [4, 5, 6]]
np.append(A, [[7, 8, 9]], axis=0)
>> array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
#or
np.r_[A,[[7,8,9]]]
Just to someone's intersted, if you would like to add a column,
array = np.c_[A,np.zeros(#A's row size)]
following what we did before on matrix A, adding a column to it
np.c_[A, [2,8]]
>> array([[1, 2, 3, 2],
[4, 5, 6, 8]])
If you want to prepend, you can just flip the order of the arguments, i.e.:
np.r_([[7, 8, 9]], A)
>> array([[7, 8, 9],
[1, 2, 3],
[4, 5, 6]])
If no calculations are necessary after every row, it's much quicker to add rows in python, then convert to numpy. Here are timing tests using python 3.6 vs. numpy 1.14, adding 100 rows, one at a time:
import numpy as np
from time import perf_counter, sleep
def time_it():
# Compare performance of two methods for adding rows to numpy array
py_array = [[0, 1, 2], [0, 2, 0]]
py_row = [4, 5, 6]
numpy_array = np.array(py_array)
numpy_row = np.array([4,5,6])
n_loops = 100
start_clock = perf_counter()
for count in range(0, n_loops):
numpy_array = np.vstack([numpy_array, numpy_row]) # 5.8 micros
duration = perf_counter() - start_clock
print('numpy 1.14 takes {:.3f} micros per row'.format(duration * 1e6 / n_loops))
start_clock = perf_counter()
for count in range(0, n_loops):
py_array.append(py_row) # .15 micros
numpy_array = np.array(py_array) # 43.9 micros
duration = perf_counter() - start_clock
print('python 3.6 takes {:.3f} micros per row'.format(duration * 1e6 / n_loops))
sleep(15)
#time_it() prints:
numpy 1.14 takes 5.971 micros per row
python 3.6 takes 0.694 micros per row
So, the simple solution to the original question, from seven years ago, is to use vstack() to add a new row after converting the row to a numpy array. But a more realistic solution should consider vstack's poor performance under those circumstances. If you don't need to run data analysis on the array after every addition, it is better to buffer the new rows to a python list of rows (a list of lists, really), and add them as a group to the numpy array using vstack() before doing any data analysis.
You can also do this:
newrow = [1,2,3]
A = numpy.concatenate((A,newrow))
import numpy as np
array_ = np.array([[1,2,3]])
add_row = np.array([[4,5,6]])
array_ = np.concatenate((array_, add_row), axis=0)
I use 'np.vstack' which is faster, EX:
import numpy as np
input_array=np.array([1,2,3])
new_row= np.array([4,5,6])
new_array=np.vstack([input_array, new_row])
I use numpy.insert(arr, i, the_object_to_be_added, axis) in order to insert object_to_be_added at the i'th row(axis=0) or column(axis=1)
import numpy as np
a = np.array([[1, 2, 3], [5, 4, 6]])
# array([[1, 2, 3],
# [5, 4, 6]])
np.insert(a, 1, [55, 66], axis=1)
# array([[ 1, 55, 2, 3],
# [ 5, 66, 4, 6]])
np.insert(a, 2, [50, 60, 70], axis=0)
# array([[ 1, 2, 3],
# [ 5, 4, 6],
# [50, 60, 70]])
Too old discussion, but I hope it helps someone.
If you can do the construction in a single operation, then something like the vstack-with-fancy-indexing answer is a fine approach. But if your condition is more complicated or your rows come in on the fly, you may want to grow the array. In fact the numpythonic way to do something like this - dynamically grow an array - is to dynamically grow a list:
A = np.array([[1,2,3],[4,5,6]])
Alist = [r for r in A]
for i in range(100):
newrow = np.arange(3)+i
if i%5:
Alist.append(newrow)
A = np.array(Alist)
del Alist
Lists are highly optimized for this kind of access pattern; you don't have convenient numpy multidimensional indexing while in list form, but for as long as you're appending it's hard to do better than a list of row arrays.
You can use numpy.append() to append a row to numpty array and reshape to a matrix later on.
import numpy as np
a = np.array([1,2])
a = np.append(a, [3,4])
print a
# [1,2,3,4]
# in your example
A = [1,2]
for row in X:
A = np.append(A, row)