Example Problem
As a simple example, consider the numpy array arr as defined below:
import numpy as np
arr = np.array([[5, np.nan, np.nan, 7, 2],
[3, np.nan, 1, 8, np.nan],
[4, 9, 6, np.nan, np.nan]])
where arr looks like this in console output:
array([[ 5., nan, nan, 7., 2.],
[ 3., nan, 1., 8., nan],
[ 4., 9., 6., nan, nan]])
I would now like to row-wise 'forward-fill' the nan values in array arr. By that I mean replacing each nan value with the nearest valid value from the left. The desired result would look like this:
array([[ 5., 5., 5., 7., 2.],
[ 3., 3., 1., 8., 8.],
[ 4., 9., 6., 6., 6.]])
Tried thus far
I've tried using for-loops:
for row_idx in range(arr.shape[0]):
for col_idx in range(arr.shape[1]):
if np.isnan(arr[row_idx][col_idx]):
arr[row_idx][col_idx] = arr[row_idx][col_idx - 1]
I've also tried using a pandas dataframe as an intermediate step (since pandas dataframes have a very neat built-in method for forward-filling):
import pandas as pd
df = pd.DataFrame(arr)
df.fillna(method='ffill', axis=1, inplace=True)
arr = df.as_matrix()
Both of the above strategies produce the desired result, but I keep on wondering: wouldn't a strategy that uses only numpy vectorized operations be the most efficient one?
Summary
Is there another more efficient way to 'forward-fill' nan values in numpy arrays? (e.g. by using numpy vectorized operations)
Update: Solutions Comparison
I've tried to time all solutions thus far. This was my setup script:
import numba as nb
import numpy as np
import pandas as pd
def random_array():
choices = [1, 2, 3, 4, 5, 6, 7, 8, 9, np.nan]
out = np.random.choice(choices, size=(1000, 10))
return out
def loops_fill(arr):
out = arr.copy()
for row_idx in range(out.shape[0]):
for col_idx in range(1, out.shape[1]):
if np.isnan(out[row_idx, col_idx]):
out[row_idx, col_idx] = out[row_idx, col_idx - 1]
return out
#nb.jit
def numba_loops_fill(arr):
'''Numba decorator solution provided by shx2.'''
out = arr.copy()
for row_idx in range(out.shape[0]):
for col_idx in range(1, out.shape[1]):
if np.isnan(out[row_idx, col_idx]):
out[row_idx, col_idx] = out[row_idx, col_idx - 1]
return out
def pandas_fill(arr):
df = pd.DataFrame(arr)
df.fillna(method='ffill', axis=1, inplace=True)
out = df.as_matrix()
return out
def numpy_fill(arr):
'''Solution provided by Divakar.'''
mask = np.isnan(arr)
idx = np.where(~mask,np.arange(mask.shape[1]),0)
np.maximum.accumulate(idx,axis=1, out=idx)
out = arr[np.arange(idx.shape[0])[:,None], idx]
return out
followed by this console input:
%timeit -n 1000 loops_fill(random_array())
%timeit -n 1000 numba_loops_fill(random_array())
%timeit -n 1000 pandas_fill(random_array())
%timeit -n 1000 numpy_fill(random_array())
resulting in this console output:
1000 loops, best of 3: 9.64 ms per loop
1000 loops, best of 3: 377 µs per loop
1000 loops, best of 3: 455 µs per loop
1000 loops, best of 3: 351 µs per loop
Here's one approach -
mask = np.isnan(arr)
idx = np.where(~mask,np.arange(mask.shape[1]),0)
np.maximum.accumulate(idx,axis=1, out=idx)
out = arr[np.arange(idx.shape[0])[:,None], idx]
If you don't want to create another array and just fill the NaNs in arr itself, replace the last step with this -
arr[mask] = arr[np.nonzero(mask)[0], idx[mask]]
Sample input, output -
In [179]: arr
Out[179]:
array([[ 5., nan, nan, 7., 2., 6., 5.],
[ 3., nan, 1., 8., nan, 5., nan],
[ 4., 9., 6., nan, nan, nan, 7.]])
In [180]: out
Out[180]:
array([[ 5., 5., 5., 7., 2., 6., 5.],
[ 3., 3., 1., 8., 8., 5., 5.],
[ 4., 9., 6., 6., 6., 6., 7.]])
Update: As pointed out by financial_physician in the comments, my initially proposed solution can simply be exchanged with ffill on the reversed array and then reversing the result. There is no relevant performance loss. My initial solution seems to be 2% or 3% faster according to %timeit. I updated the code example below but left my initial text as it was.
For those that came here looking for the backward-fill of NaN values, I modified the solution provided by Divakar above to do exactly that. The trick is that you have to do the accumulation on the reversed array using the minimum except for the maximum.
Here is the code:
# ffill along axis 1, as provided in the answer by Divakar
def ffill(arr):
mask = np.isnan(arr)
idx = np.where(~mask, np.arange(mask.shape[1]), 0)
np.maximum.accumulate(idx, axis=1, out=idx)
out = arr[np.arange(idx.shape[0])[:,None], idx]
return out
# Simple solution for bfill provided by financial_physician in comment below
def bfill(arr):
return ffill(arr[:, ::-1])[:, ::-1]
# My outdated modification of Divakar's answer to do a backward-fill
def bfill_old(arr):
mask = np.isnan(arr)
idx = np.where(~mask, np.arange(mask.shape[1]), mask.shape[1] - 1)
idx = np.minimum.accumulate(idx[:, ::-1], axis=1)[:, ::-1]
out = arr[np.arange(idx.shape[0])[:,None], idx]
return out
# Test both functions
arr = np.array([[5, np.nan, np.nan, 7, 2],
[3, np.nan, 1, 8, np.nan],
[4, 9, 6, np.nan, np.nan]])
print('Array:')
print(arr)
print('\nffill')
print(ffill(arr))
print('\nbfill')
print(bfill(arr))
Output:
Array:
[[ 5. nan nan 7. 2.]
[ 3. nan 1. 8. nan]
[ 4. 9. 6. nan nan]]
ffill
[[5. 5. 5. 7. 2.]
[3. 3. 1. 8. 8.]
[4. 9. 6. 6. 6.]]
bfill
[[ 5. 7. 7. 7. 2.]
[ 3. 1. 1. 8. nan]
[ 4. 9. 6. nan nan]]
Edit: Update according to comment of MS_
I liked Divakar's answer on pure numpy.
Here's a generalized function for n-dimensional arrays:
def np_ffill(arr, axis):
idx_shape = tuple([slice(None)] + [np.newaxis] * (len(arr.shape) - axis - 1))
idx = np.where(~np.isnan(arr), np.arange(arr.shape[axis])[idx_shape], 0)
np.maximum.accumulate(idx, axis=axis, out=idx)
slc = [np.arange(k)[tuple([slice(None) if dim==i else np.newaxis
for dim in range(len(arr.shape))])]
for i, k in enumerate(arr.shape)]
slc[axis] = idx
return arr[tuple(slc)]
AFIK pandas can only work with two dimensions, despite having multi-index to make up for it. The only way to accomplish this would be to flatten a DataFrame, unstack desired level, restack, and finally reshape as original. This unstacking/restacking/reshaping, with the pandas sorting involved, is just unnecessary overhead to achieve the same result.
Testing:
def random_array(shape):
choices = [1, 2, 3, 4, np.nan]
out = np.random.choice(choices, size=shape)
return out
ra = random_array((2, 4, 8))
print('arr')
print(ra)
print('\nffull')
print(np_ffill(ra, 1))
raise SystemExit
Output:
arr
[[[ 3. nan 4. 1. 4. 2. 2. 3.]
[ 2. nan 1. 3. nan 4. 4. 3.]
[ 3. 2. nan 4. nan nan 3. 4.]
[ 2. 2. 2. nan 1. 1. nan 2.]]
[[ 2. 3. 2. nan 3. 3. 3. 3.]
[ 3. 3. 1. 4. 1. 4. 1. nan]
[ 4. 2. nan 4. 4. 3. nan 4.]
[ 2. 4. 2. 1. 4. 1. 3. nan]]]
ffull
[[[ 3. nan 4. 1. 4. 2. 2. 3.]
[ 2. nan 1. 3. 4. 4. 4. 3.]
[ 3. 2. 1. 4. 4. 4. 3. 4.]
[ 2. 2. 2. 4. 1. 1. 3. 2.]]
[[ 2. 3. 2. nan 3. 3. 3. 3.]
[ 3. 3. 1. 4. 1. 4. 1. 3.]
[ 4. 2. 1. 4. 4. 3. 1. 4.]
[ 2. 4. 2. 1. 4. 1. 3. 4.]]]
Use Numba. This should give a significant speedup:
import numba
#numba.jit
def loops_fill(arr):
...
I like Divakar's answer, but it doesn't work for an edge case where a row starts with np.nan, like the arr below
arr = np.array([[9, np.nan, 4, np.nan, 6, 6, 7, 2, 3, np.nan],
[ np.nan, 5, 5, 6, 5, 3, 2, 1, np.nan, 10]])
The output using Divakar's code would be:
[[ 9. 9. 4. 4. 6. 6. 7. 2. 3. 3.]
[nan 4. 5. 6. 5. 3. 2. 1. 1. 10.]]
Divakar's code can be simplified a bit, and the simplified version solves this issue at the same time:
arr[np.isnan(arr)] = arr[np.nonzero(np.isnan(arr))[0], np.nonzero(np.isnan(arr))[1]-1]
In case of several np.nans in a row (either in the beginning or in the middle), just repeat this operation several times. For instance, if the array has 5 consecutive np.nans, the following code will "forward fill" all of them with the number before these np.nans:
for i in range(0, 5):
value[np.isnan(value)] = value[np.nonzero(np.isnan(value))[0], np.nonzero(np.isnan(value))[1]-1]
For those who are interested in the problem of having leading np.nan after foward-filling, the following works:
mask = np.isnan(arr)
first_non_zero_idx = (~mask!=0).argmax(axis=1) #Get indices of first non-zero values
arr = [ np.hstack([
[arr[i,first_nonzero]]*(first_nonzero),
arr[i,first_nonzero:]])
for i, first_nonzero in enumerate(first_non_zero_idx) ]
bottleneck push function is a good option to forward fill. It's normally used internally in packages like Xarray, it should be faster than other alternatives and the package also has a set of benchmarks.
Example:
import numpy as np
from bottleneck import push
a = np.array(
[
[1, np.nan, 3],
[np.nan, 3, 2],
[2, np.nan, np.nan]
]
)
push(a, axis=0)
array([[ 1., nan, 3.],
[ 1., 3., 2.],
[ 2., 3., 2.]])
Use bottleneck module, it comes along with pandas or numpy module so no need to separately install.
Below code should give you desired result.
import bottleneck as bn
bn.push(arr,axis=1)
If you're willing to use Pandas/ xarray: Let axis be the direction you wish to ffill/bfill over, as shown below,
xr.DataArray(arr).ffill(f'dim_{axis}').values
xr.DataArray(arr).bfill(f'dim_{axis}').values
More information:
http://xarray.pydata.org/en/stable/generated/xarray.DataArray.ffill.html
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.ffill.html
One liner:
result = np.where(np.isnan(arr), 0, arr)
In a function with forcing float (I needed it in my case because I had dtype=object).
def fillna(arr):
arr = np.array(arr,dtype=float)
out = np.where(np.isnan(arr), 0, arr)
return out
arr = np.array([[5, np.nan, np.nan, 7, 2],
[3, np.nan, 1, 8, np.nan],
[4, 9, 6, np.nan, np.nan]])
result = fillna(arr)
print(result)
# result
# array([[5., 0., 0., 7., 2.],
# [3., 0., 1., 8., 0.],
# [4., 9., 6., 0., 0.]])
unless I miss something, the solutions does not works on any example:
arr = np.array([[ 3.],
[ 8.],
[np.nan],
[ 7.],
[np.nan],
[ 1.],
[np.nan],
[ 3.],
[ 8.],
[ 8.]])
print("A:::: \n", arr)
print("numpy_fill::: \n ", numpy_fill(arr))
print("loop_fill", loops_fill(arr))
A::::
[[ 3.]
[ 8.]
[nan]
[ 7.]
[nan]
[ 1.]
[nan]
[ 3.]
[ 8.]
[ 8.]]
numpy_fill:::
[[ 3.]
[ 8.]
[nan]
[ 7.]
[nan]
[ 1.]
[nan]
[ 3.]
[ 8.]
[ 8.]]
loop_fill [[ 3.]
[ 8.]
[nan]
[ 7.]
[nan]
[ 1.]
[nan]
[ 3.]
[ 8.]
[ 8.]]
Comments ??
Minor improvement of of RichieV generalized pure numpy solution with axis selection and 'backward' support
def _np_fill_(arr, axis=-1, fill_dir='f'):
"""Base function for np_fill, np_ffill, np_bfill."""
if axis < 0:
axis = len(arr.shape) + axis
if fill_dir.lower() in ['b', 'backward']:
dir_change = tuple([*[slice(None)]*axis, slice(None, None, -1)])
return np_ffill(arr[dir_change])[dir_change]
elif fill_dir.lower() not in ['f', 'forward']:
raise KeyError(f"fill_dir must be one of: 'b', 'backward', 'f', 'forward'. Got: {fill_dir}")
idx_shape = tuple([slice(None)] + [np.newaxis] * (len(arr.shape) - axis - 1))
idx = np.where(~np.isnan(arr), np.arange(arr.shape[axis])[idx_shape], 0)
np.maximum.accumulate(idx, axis=axis, out=idx)
slc = [np.arange(k)[tuple([slice(None) if dim==i else np.newaxis
for dim in range(len(arr.shape))])]
for i, k in enumerate(arr.shape)]
slc[axis] = idx
return arr[tuple(slc)]
def np_fill(arr, axis=-1, fill_dir='f'):
"""General fill function which supports multiple filling steps. I.e.:
fill_dir=['f', 'b'] or fill_dir=['b', 'f']"""
if isinstance(fill_dir, (tuple, list, np.ndarray)):
for i in fill_dir:
arr = _np_fill_(arr, axis=axis, fill_dir=i)
else:
arr = _np_fill_(arr, axis=axis, fill_dir=fill_dir)
return arr
def np_ffill(arr, axis=-1):
return np_fill(arr, axis=axis, fill_dir='forward')
def np_bfill(arr, axis=-1):
return np_fill(arr, axis=axis, fill_dir='backward')
I used np.nan_to_num
Example:
data = np.nan_to_num(data, data.mean())
Reference : Numpy document
Related
Consider the code below:
import tensorflow as tf
test=tf.constant([[[100., 2., -30.],[-4,5,6]], [[4., 5., 6.],[-7,8,9]]]) # matrix
print(test)
test1=tf.constant([[[100.]],[[ 8.]]])
print(test1)
When adding test1 to the first two columns of test we would get the following output:
print(test[:,:,0:2]+test1)
I do not want to add test1 variable to the last column of the test variable, but at the same time I would like to include the last column of the test variable in the output unchanged:
[[[200. 102., -30.]
[ 96. 105., 6.]]
[[ 12. 13., 6]
[ 1. 16., 9]]]
How would I code this quickly?
The simplest option would be to just use tf.concat:
import tensorflow as tf
test = tf.constant([[[100., 2., -30.],[-4,5,6]], [[4., 5., 6.],[-7,8,9]]]) # matrix
test1 = tf.constant([[[100.]],[[ 8.]]])
print(tf.concat([test[:,:,0:2] + test1, test[:,:,2:]], axis=-1))
tf.Tensor(
[[[200. 102. -30.]
[ 96. 105. 6.]]
[[ 12. 13. 6.]
[ 1. 16. 9.]]], shape=(2, 2, 3), dtype=float32)
Please let me know if anyone knows of a better way to do about the following.
I am trying to replace some values in numpy array.
Replace condition is differ in each columns.
Suppose I have a numpy array and list of nodata values like:
import numpy as np
array = np.array([[ 1, 2, 3],
[ 4, 5, 6],
[ 7, 8, 9],
[10,11,12]])
nodata_values = [4, 8, 3]
and what I want to have is array which values are replaced like
array([[ 1., 2., nan],
[nan, 5., 6.],
[ 7., nan, 9.],
[10., 11., 12.]])
I know I can do this:
np.array([np.where(i == nodata_values[idx], np.nan, i)
for idx, i in enumerate(array.T)]).T
But this code using for loop inside therefore applying it to a table with tens of thousands of rows will takes time.
Use np.isin to create boolean index & broadcast.
astype to avoid ValueError: cannot convert float NaN to integer
import numpy as np
array = array.astype(np.float)
array[np.isin(array , nodata_values)] = np.NaN
[[ 1. 2. nan]
[nan 5. 6.]
[ 7. nan 9.]
[10. 11. 12.]]
I've been trying to assign values from an array to another array, specifically from an array with values to a zeros array. Position of these values in the zeros array is also very essential. This is also a small piece of a bigger code, the bigger picture is to be able to import values from an excel spreadsheet into a zeros matrix. This is my problem:
import numpy as np
x = np.zeros((2,3))
P= np.asarray ([1,2,3,4,5,6])
for i in range(0,2):
for j in range(0,3):
x[i,j] = P[(i-1)*3+j] # 3 is the counter in x direction, nx
x
With this code, the output is (which is what I want):
array([[4., 5., 6.],
[1., 2., 3.]])
However if I try to the expand the array, as such:
import numpy as np
x = np.zeros((3,3))
P= np.asarray ([1,2,3,4,5,6,7,8,9])
for i in range(0,3):
for j in range(0,3):
x[i,j] = P[(i-1)*3+j] # 3 is the counter in x direction, nx
x
The output is:
array([[7., 8., 9.],
[1., 2., 3.],
[4., 5., 6.]])
I expect the output to be:
array([[7., 8., 9.],
[4., 5., 6.],
[1., 2., 3.]])
Is there a reason why the ouput is changing with the expansion of the array?
You don't need to iterate:
In [323]: P=np.arange(1,10).reshape(3,3)[::-1,:]
In [324]: P
Out[324]:
array([[7, 8, 9],
[4, 5, 6],
[1, 2, 3]])
As for your loop, look a the i,j's:
In [325]: for i in range(3):
...: for j in range(3):
...: print(i,j,(i-1)*3+j)
...:
0 0 -3
0 1 -2
0 2 -1
1 0 0
1 1 1
1 2 2
2 0 3
2 1 4
2 2 5
You don't need to use loop, just use flip() with reshape().
import numpy as np
m = 3 # number of rows you want
n = 3 # number of column you want
P = np.asarray ([1,2,3,4,5,6,7,8,9])
P = np.flip(P.reshape(m,n), axis=0)
print(P)
[[7 8 9]
[4 5 6]
[1 2 3]]
If you want to assign it to a zero matrix, you can just iterate through the indices.
For example, let's say you have a much bigger zero matrix, you want to fill row x, y, z with the current matrix generated.
zero = np.zeros((10, 3))
print(zero.shape)
zero[[2, 5, 7], : ] = P # randomly assigning P to index 2, 5, 7th row of zero matrix
print(zero)
(10, 3)
[[0. 0. 0.]
[0. 0. 0.]
[7. 8. 9.]
[0. 0. 0.]
[0. 0. 0.]
[4. 5. 6.]
[0. 0. 0.]
[1. 2. 3.]
[0. 0. 0.]
[0. 0. 0.]]
You can also loop through:
for i in range(3):
zero[i,:] = P[i,:]
Hi there i got a Matrix like this
A=[[nan, 4, nan],[3 , 7 , 8],[nan, 23, nan]]
and I would like to get a mask from the the Matrix A, that is as follows
mask=[[nan, 0, nan],[0, 0, 0],[nan, 0, nan]]
for that I have tried:
import numpy as np
A=[[nan, 4, nan],[3 , 7 , 8],[nan, 23, nan]]
mask=A
mask[np.isfinite(A)]=0
But this also deletes the numerical values of the Matrix A.
You need to do a copy of A in order to keep the values in A, see: https://docs.python.org/2/library/copy.html
In your case this would be
A=[[nan, 4, nan],[3 , 7 , 8],[nan, 23, nan]]
mask=np.array(A.copy())
mask[~np.isnan(A)] = 0
You could use a masked array, in order to mask those values that aren't np.nan, and fill the masked array with 0:
A = np.array([[np.nan, 4, np.nan],[3 , 7 , 8],[np.nan, 23, np.nan]])
np.ma.masked_array(A, mask = ~np.isnan(A)).filled(0)
array([[nan, 0., nan],
[ 0., 0., 0.],
[nan, 0., nan]])
Using A[~np.isnan(A)]:
from numpy import *
A=[[NaN, 4, NaN],[3 , 7 , 8],[NaN, 23, NaN]]
A = np.array(A)
A[~np.isnan(A)] = 0
print(A)
OUTPUT:
[[nan 0. nan]
[ 0. 0. 0.]
[nan 0. nan]]
I'm having trouble understanding a basic concept with tensorflow. How does indexing work for tensor read/write operations? In order to make this specific, how can the following numpy examples be translated to tensorflow (using tensors for the arrays, indices and values being assigned):
x = np.zeros((3, 4))
row_indices = np.array([1, 1, 2])
col_indices = np.array([0, 2, 3])
x[row_indices, col_indices] = 2
x
with output:
array([[ 0., 0., 0., 0.],
[ 2., 0., 2., 0.],
[ 0., 0., 0., 2.]])
... and ...
x[row_indices, col_indices] = np.array([5, 4, 3])
x
with output:
array([[ 0., 0., 0., 0.],
[ 5., 0., 4., 0.],
[ 0., 0., 0., 3.]])
... and finally ...
y = x[row_indices, col_indices]
y
with output:
array([ 5., 4., 3.])
There's github issue #206 to support this nicely, meanwhile you have to resort to verbose work-arounds
The first example can be done with tf.select that combines two same-shaped tensors by selecting each element from one or the other
tf.reset_default_graph()
row_indices = tf.constant([1, 1, 2])
col_indices = tf.constant([0, 2, 3])
x = tf.zeros((3, 4))
sess = tf.InteractiveSession()
# get list of ((row1, col1), (row2, col2), ..)
coords = tf.transpose(tf.pack([row_indices, col_indices]))
# get tensor with 1's at positions (row1, col1),...
binary_mask = tf.sparse_to_dense(coords, x.get_shape(), 1)
# convert 1/0 to True/False
binary_mask = tf.cast(binary_mask, tf.bool)
twos = 2*tf.ones(x.get_shape())
# make new x out of old values or 2, depending on mask
x = tf.select(binary_mask, twos, x)
print x.eval()
gives
[[ 0. 0. 0. 0.]
[ 2. 0. 2. 0.]
[ 0. 0. 0. 2.]]
The second one could be done with scatter_update, except scatter_update only supports on linear indices and works on variables. So you could create a temporary variable and use reshaping like this. (to avoid variables you could use dynamic_stitch, see the end)
# get linear indices
linear_indices = row_indices*x.get_shape()[1]+col_indices
# turn 'x' into 1d variable since "scatter_update" supports linear indexing only
x_flat = tf.Variable(tf.reshape(x, [-1]))
# no automatic promotion, so make updates float32 to match x
updates = tf.constant([5, 4, 3], dtype=tf.float32)
sess.run(tf.initialize_all_variables())
sess.run(tf.scatter_update(x_flat, linear_indices, updates))
# convert back into original shape
x = tf.reshape(x_flat, x.get_shape())
print x.eval()
gives
[[ 0. 0. 0. 0.]
[ 5. 0. 4. 0.]
[ 0. 0. 0. 3.]]
Finally the third example is already supported with gather_nd, you write
print tf.gather_nd(x, coords).eval()
To get
[ 5. 4. 3.]
Edit, May 6
The update x[cols,rows]=newvals can be done without using Variables (which occupy memory between session run calls) by using select with sparse_to_dense that takes vector of sparse values, or relying on dynamic_stitch
sess = tf.InteractiveSession()
x = tf.zeros((3, 4))
row_indices = tf.constant([1, 1, 2])
col_indices = tf.constant([0, 2, 3])
# no automatic promotion, so specify float type
replacement_vals = tf.constant([5, 4, 3], dtype=tf.float32)
# convert to linear indexing in row-major form
linear_indices = row_indices*x.get_shape()[1]+col_indices
x_flat = tf.reshape(x, [-1])
# use dynamic stitch, it merges the array by taking value either
# from array1[index1] or array2[index2], if indices conflict,
# the later one is used
unchanged_indices = tf.range(tf.size(x_flat))
changed_indices = linear_indices
x_flat = tf.dynamic_stitch([unchanged_indices, changed_indices],
[x_flat, replacement_vals])
x = tf.reshape(x_flat, x.get_shape())
print x.eval()