Develop Reference

Python program to delete all the rows and columns with all zeros and print the remaining matrix

Is there any program possible with a complexity less than O(mn)? The input is in the form as the first line contains MN and the next M lines each containing N integers
For example
4 4
1 0 3 4
0 0 0 0
4 0 6 8
4 0 2 4
The output should be:
1 3 4
4 6 8
4 2 4

You can do this by individually filtering rows and columns with all values equal to 1 but checking if set(row or column)!={0}
arr = [[1, 0, 3, 4],
[0, 0, 0, 0],
[4, 0, 6, 8],
[4, 0, 2, 4]]
rows = [i for i in arr if set(i)!={0}]
cols = [i for i in zip(*rows) if set(i)!={0}]
arr_new = [list(i) for i in zip(*cols)]
print(arr_new)
[[1, 3, 4],
[4, 6, 8],
[4, 2, 4]]
EDIT:
If you are ok with using numpy then you can do this a bit more easily -
import numpy as np
arr = np.array(arr)
arr[~(arr==0).all(0)][:,~(arr==0).all(1)]
array([[1, 3, 4],
[4, 6, 8],
[4, 2, 4]])

How to make a histogram from all of the values in a dataframe?

Suppose a dataframe like the one below:
a b c d e f g
b 0 0 1 2 0 5
c 1 3 2 0 2 5
d 12 0 0 1 3 9
e 3 4 7 8 9 0
f 0 0 0 0 1 1
g 3 4 4 5 1 0
I want to generate a histogram of the frequencies each value (on each cell) appears on the dataframe.
Note that on my actual dataframe I have a matrix of 260 rows by 260 columns with values ranging from 0 to 30, including floating points.
Any help would be extremely appreciated :)

This will make a histogram from all of the values:
import matplotlib.pyplot as plt
import pandas as pd
df = pd.DataFrame([
[0, 0, 1, 2, 0, 5],
[1, 3, 2, 0, 2, 5],
[12, 0, 0, 1, 3, 9],
[3, 4, 7, 8, 9, 0],
[0, 0, 0, 0, 1, 1],
[3, 4, 4, 5, 1, 0],
])
plt.hist(df.to_numpy().flatten())
plt.show()
Alternatively you can use:
df.stack().hist(grid=False)
plt.show()
to get the exact same result.

Writing a 3d numpy array that is readable in matlab

I'm trying to save a 3D numpy array to my disk so that I can later read it in matlab. I've had some difficulty using numpy.savetxt() on a 3D array, so my solution has been to first convert it to a 1D array using the following code:
import numpy
array = numpy.array([[0, 1, 2, 3],
[0, 1, 1, 3],
[3, 1, 3, 1]])
ndarray = numpy.dstack((array, array, array))
darray = ndarray.reshape(36,1)
numpy.savetxt('test.txt', darray, fmt = '%i')
Then in matlab it can be read with the following code:
file = fopen('test.txt')
array = fscanf(file, '%f')
My issue now is converting it back to the original shape. Using reshape(array, 3,4,3) yields the following:
ans(:,:,1) =
0 1 2 3
0 1 2 3
0 1 2 3
ans(:,:,2) =
0 1 1 3
0 1 1 3
0 1 1 3
ans(:,:,3) =
3 1 3 1
3 1 3 1
3 1 3 1
I've tried to transpose the 1D matlab array, then use reshape() but get the same array.
What matlab function can I apply to achieve my original python array?

You want to permute the dimensions. In numpy this is transpose. There are two complications - the 'F' order of MATLAB matrices, and the display pattern, using blocks on the last dimension (which is the outer one with F order). Jump to the end of this answer for details.
===
In [72]: arr = np.array([[0, 1, 2, 3],
...: [0, 1, 1, 3],
...: [3, 1, 3, 1]])
...:
In [80]: np.dstack((arr,arr+1))
Out[80]:
array([[[0, 1],
[1, 2],
[2, 3],
[3, 4]],
[[0, 1],
[1, 2],
[1, 2],
[3, 4]],
[[3, 4],
[1, 2],
[3, 4],
[1, 2]]])
In [81]: np.dstack((arr,arr+1)).shape
Out[81]: (3, 4, 2)
In [75]: from scipy.io import loadmat, savemat
In [76]: pwd
Out[76]: '/home/paul/mypy'
In [83]: savemat('test3',{'arr':arr, 'arr3':arr3})
In Octave
>> load 'test3.mat'
>> arr
arr =
0 1 2 3
0 1 1 3
3 1 3 1
>> arr3
arr3 =
ans(:,:,1) =
0 1 2 3
0 1 1 3
3 1 3 1
ans(:,:,2) =
1 2 3 4
1 2 2 4
4 2 4 2
>> size(arr3)
ans =
3 4 2
back in numpy I can display the array as 2 3x4 blocks with:
In [95]: arr3[:,:,0]
Out[95]:
array([[0, 1, 2, 3],
[0, 1, 1, 3],
[3, 1, 3, 1]])
In [96]: arr3[:,:,1]
Out[96]:
array([[1, 2, 3, 4],
[1, 2, 2, 4],
[4, 2, 4, 2]])
These arrays, ravelled to 1d (showing in effect the layout of values in the underlying databuffer):
In [100]: arr.ravel()
Out[100]: array([0, 1, 2, 3, 0, 1, 1, 3, 3, 1, 3, 1])
In [101]: arr3.ravel()
Out[101]:
array([0, 1, 1, 2, 2, 3, 3, 4, 0, 1, 1, 2, 1, 2, 3, 4, 3, 4, 1, 2, 3, 4, 1, 2])
The corresponding ravel in Octave:
>> arr(:).'
ans =
0 0 3 1 1 1 2 1 3 3 3 1
>> arr3(:).'
ans =
0 0 3 1 1 1 2 1 3 3 3 1 1 1 4 2 2 2 3 2 4 4 4 2
MATLAB uses F (fortran) order, with the first dimension changing fastest. Thus it is natural to display blocks arr(:,:i). You can specify order='F' when creating and working with numpy arrays. But it can be tricky keeping the order straight, especially when working with 3d. loadmat/savemat try to do some of the reordering for us. For example a 2d MATLAB matrix loads as an order F array in numpy.
In [107]: np.array([0,0,3,1,1,1,2,1,3,3,3,1])
Out[107]: array([0, 0, 3, 1, 1, 1, 2, 1, 3, 3, 3, 1])
In [108]: np.array([0,0,3,1,1,1,2,1,3,3,3,1]).reshape(4,3)
Out[108]:
array([[0, 0, 3],
[1, 1, 1],
[2, 1, 3],
[3, 3, 1]])
In [109]: np.array([0,0,3,1,1,1,2,1,3,3,3,1]).reshape(4,3).T
Out[109]:
array([[0, 1, 2, 3],
[0, 1, 1, 3],
[3, 1, 3, 1]])
In [111]: np.array([0,0,3,1,1,1,2,1,3,3,3,1]).reshape((3,4),order='F')
Out[111]:
array([[0, 1, 2, 3],
[0, 1, 1, 3],
[3, 1, 3, 1]])
It might easier to keep track of shapes with this array:
In [112]: arr3 = np.arange(2*3*4).reshape(2,3,4)
In [113]: arr3f = np.arange(2*3*4).reshape(2,3,4, order='F')
In [114]: arr3
Out[114]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]])
In [115]: arr3f
Out[115]:
array([[[ 0, 6, 12, 18],
[ 2, 8, 14, 20],
[ 4, 10, 16, 22]],
[[ 1, 7, 13, 19],
[ 3, 9, 15, 21],
[ 5, 11, 17, 23]]])
In [116]: arr3f.ravel()
Out[116]:
array([ 0, 6, 12, 18, 2, 8, 14, 20, 4, 10, 16, 22, 1, 7, 13, 19, 3,
9, 15, 21, 5, 11, 17, 23])
In [117]: arr3f.ravel(order='F')
Out[117]:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23])
In [118]: savemat('test3',{'arr3':arr3, 'arr3f':arr3f})
In Octave:
>> arr3
arr3 =
ans(:,:,1) =
0 4 8
12 16 20
ans(:,:,2) =
1 5 9
13 17 21
....
>> arr3f
arr3f =
ans(:,:,1) =
0 2 4
1 3 5
ans(:,:,2) =
6 8 10
7 9 11
...
>> arr3.ravel()'
error: int32 matrix cannot be indexed with .
>> arr3(:)'
ans =
Columns 1 through 20:
0 12 4 16 8 20 1 13 5 17 9 21 2 14 6 18 10 22 3 15
Columns 21 through 24:
7 19 11 23
>> arr3f(:)'
ans =
Columns 1 through 20:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Columns 21 through 24:
20 21 22 23
arr3f still looks 'messedup' when printed by blocks, but when raveled we see that values are in same F order. That's also evident if we print the last 'block' of the numpy array:
In [119]: arr3f[:,:,0]
Out[119]:
array([[0, 2, 4],
[1, 3, 5]])
So to match up numpy and matlab we have to keep 2 things straight - the order, and the block display style.
My MATLAB is rusty, but I found permute with is similar to the np.transpose. Using that to reorder the dimensions:
>> permute(arr3,[3,2,1])
ans =
ans(:,:,1) =
0 4 8
1 5 9
2 6 10
3 7 11
ans(:,:,2) =
12 16 20
13 17 21
14 18 22
15 19 23
>> permute(arr3,[3,2,1])(:)'
ans =
Columns 1 through 20:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Columns 21 through 24:
20 21 22 23
The equivalent transpose in numpy
In [121]: arr3f.transpose(2,1,0).ravel()
Out[121]:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23])
(Sorry for the rambling answer. I may go back an edit it. Hopefully it gives you something to work with.)
===
Let's try to apply that rambling more explicitly to your case
In [122]: x = np.array([[0, 1, 2, 3],
...: [0, 1, 1, 3],
...: [3, 1, 3, 1]])
...:
In [123]: x3 = np.dstack((x,x,x))
In [125]: dx3 = x3.reshape(36,1)
In [126]: np.savetxt('test3.txt',dx3, fmt='%i')
In [127]: cat test3.txt
0
0
0
....
3
3
1
1
1
In Octave
>> file = fopen('test3.txt')
file = 21
>> array = fscanf(file,'%f')
array =
0
0
....
>> reshape(array,3,4,3)
ans =
ans(:,:,1) =
0 1 2 3
0 1 2 3
0 1 2 3
ans(:,:,2) =
0 1 1 3
0 1 1 3
0 1 1 3
ans(:,:,3) =
3 1 3 1
3 1 3 1
3 1 3 1
and with the perumtation
>> permute(reshape(array,3,4,3),[3,2,1])
ans =
ans(:,:,1) =
0 1 2 3
0 1 1 3
3 1 3 1
ans(:,:,2) =
0 1 2 3
0 1 1 3
3 1 3 1
ans(:,:,3) =
0 1 2 3
0 1 1 3
3 1 3 1

numpy.tile did not work as Matlab repmat

According to What is the equivalent of MATLAB's repmat in NumPy, I tried to build 3x3x5 array from 3x3 array using python.
In Matlab, this work as I expected.
a = [1,1,1;1,2,1;1,1,1];
a_= repmat(a,[1,1,5]);
size(a_) = 3 3 5
But for numpy.tile
b = numpy.array([[1,1,1],[1,2,1],[1,1,1]])
b_ = numpy.tile(b, [1,1,5])
b_.shape = (1, 3, 15)
If I want to generate the same array as in Matlab, what is the equivalent?
Edit 1
The output I would expect to get is
b_(:,:,1) =
1 1 1
1 2 1
1 1 1
b_(:,:,2) =
1 1 1
1 2 1
1 1 1
b_(:,:,3) =
1 1 1
1 2 1
1 1 1
b_(:,:,4) =
1 1 1
1 2 1
1 1 1
b_(:,:,5) =
1 1 1
1 2 1
1 1 1
but what #farenorth and the numpy.dstack give is
[[[1 1 1 1 1]
[1 1 1 1 1]
[1 1 1 1 1]]
[[1 1 1 1 1]
[2 2 2 2 2]
[1 1 1 1 1]]
[[1 1 1 1 1]
[1 1 1 1 1]
[1 1 1 1 1]]]

NumPy functions are not, in general, 'drop-in' replacements for matlab functions. Often times there are subtle difference to how the 'equivalent' functions are used. It does take time to adapt, but I've found the transition to be very worthwhile.
In this case, the np.tile documentation indicates what happens when you are trying to tile an array to higher dimensions than it is defined,
numpy.tile(A, reps)
Construct an array by repeating A the number of times given by reps.
If reps has length d, the result will have dimension of max(d, A.ndim).
If A.ndim < d, A is promoted to be d-dimensional by prepending new axes. So a shape (3,) array is promoted to (1, 3) for 2-D replication, or shape (1, 1, 3) for 3-D replication. If this is not the desired behavior, promote A to d-dimensions manually before calling this function.
In this case, your array is being cast to a shape of [1, 3, 3], then being tiled. So, to get your desired behavior just be sure to append a new singleton-dimension to the array where you want it,
>>> b_ = numpy.tile(b[..., None], [1, 1, 5])
>>> print(b_.shape)
(3, 3, 5)
Note here that I've used None (i.e. np.newaxis) and ellipses notation to specify a new dimension at the end of the array. You can find out more about these capabilities here.
Another option, which is inspired by the OP's comment would be:
b_ = np.dstack((b, ) * 5)
In this case, I've used tuple multiplication to 'repmat' the array, which is then constructed by np.dstack.
As #hpaulj indicated, Matlab and NumPy display matrices differently. To replicate the Matlab output you can do something like:
>>> for idx in xrange(b_.shape[2]):
... print 'b_[:, :, {}] = \n{}\n'.format(idx, str(b_[:, :, idx]))
...
b_[:, :, 0] =
[[1 1 1]
[1 2 1]
[1 1 1]]
b_[:, :, 1] =
[[1 1 1]
[1 2 1]
[1 1 1]]
b_[:, :, 2] =
[[1 1 1]
[1 2 1]
[1 1 1]]
b_[:, :, 3] =
[[1 1 1]
[1 2 1]
[1 1 1]]
b_[:, :, 4] =
[[1 1 1]
[1 2 1]
[1 1 1]]
Good luck!

Let's try the comparison, taking care to diversify the shapes and values.
octave:7> a=reshape(0:11,3,4)
a =
0 3 6 9
1 4 7 10
2 5 8 11
octave:8> repmat(a,[1,1,2])
ans =
ans(:,:,1) =
0 3 6 9
1 4 7 10
2 5 8 11
ans(:,:,2) =
0 3 6 9
1 4 7 10
2 5 8 11
numpy equivalent - more or less:
In [61]: a=np.arange(12).reshape(3,4)
In [62]: np.tile(a,[2,1,1])
Out[62]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]]])
numpy again, but with order F to better match the MATLAB Fortran-derived layout
In [63]: a=np.arange(12).reshape(3,4,order='F')
In [64]: np.tile(a,[2,1,1])
Out[64]:
array([[[ 0, 3, 6, 9],
[ 1, 4, 7, 10],
[ 2, 5, 8, 11]],
[[ 0, 3, 6, 9],
[ 1, 4, 7, 10],
[ 2, 5, 8, 11]]])
I'm adding the new numpy dimension at the start, because in many ways it better replicates the MATLAB practice of adding it at the end.
Try adding the new dimension at the end. The shape is (3,4,5), but you might not like the display.
np.tile(a[:,:,None],[1,1,2])
Another consideration - what happens when you flatten the tile?
octave:10> repmat(a,[1,1,2])(:).'
ans =
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
with the order F a
In [78]: np.tile(a[:,:,None],[1,1,2]).flatten()
Out[78]:
array([ 0, 0, 3, 3, 6, 6, 9, 9, 1, 1, 4, 4, 7, 7, 10, 10, 2,
2, 5, 5, 8, 8, 11, 11])
In [79]: np.tile(a,[2,1,1]).flatten()
Out[79]:
array([ 0, 3, 6, 9, 1, 4, 7, 10, 2, 5, 8, 11, 0, 3, 6, 9, 1,
4, 7, 10, 2, 5, 8, 11])
with a C order array:
In [80]: a=np.arange(12).reshape(3,4)
In [81]: np.tile(a,[2,1,1]).flatten()
Out[81]:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4,
5, 6, 7, 8, 9, 10, 11])
This last one matches the Octave layout.
So does:
In [83]: a=np.arange(12).reshape(3,4,order='F')
In [84]: np.tile(a[:,:,None],[1,1,2]).flatten(order='F')
Out[84]:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4,
5, 6, 7, 8, 9, 10, 11])
Confused yet?

In numpy, how to efficiently list all fixed-size submatrices?

I have an arbitrary NxM matrix, for example:
1 2 3 4 5 6
7 8 9 0 1 2
3 4 5 6 7 8
9 0 1 2 3 4
I want to get a list of all 3x3 submatrices in this matrix:
1 2 3 2 3 4 0 1 2
7 8 9 ; 8 9 0 ; ... ; 6 7 8
3 4 5 4 5 6 2 3 4
I can do this with two nested loops:
rows, cols = input_matrix.shape
patches = []
for row in np.arange(0, rows - 3):
for col in np.arange(0, cols - 3):
patches.append(input_matrix[row:row+3, col:col+3])
But for a large input matrix, this is slow. Is there a way to do this faster with numpy?
I've looked at np.split, but that gives me non-overlapping sub-matrices, whereas I want all possible submatrices, regardless of overlap.

You want a windowed view:
from numpy.lib.stride_tricks import as_strided
arr = np.arange(1, 25).reshape(4, 6) % 10
sub_shape = (3, 3)
view_shape = tuple(np.subtract(arr.shape, sub_shape) + 1) + sub_shape
arr_view = as_strided(arr, view_shape, arr.strides * 2
arr_view = arr_view.reshape((-1,) + sub_shape)
>>> arr_view
array([[[[1, 2, 3],
[7, 8, 9],
[3, 4, 5]],
[[2, 3, 4],
[8, 9, 0],
[4, 5, 6]],
...
[[9, 0, 1],
[5, 6, 7],
[1, 2, 3]],
[[0, 1, 2],
[6, 7, 8],
[2, 3, 4]]]])
The good part of doing it like this is that you are not copying any data, you are simply accessing the data of your original array in a different way. For large arrays this can result in tremendous memory savings.