Related
I have the following code:
import torch
d = 2
n = 50
X = torch.randn(n,d)
z = torch.tensor([[-1.0], [2.0]])
y = X # z
X.size()
z.size()
y.size()
The output is:
torch.Size([50, 2])
torch.Size([2, 1])
torch.Size([50, 1])
My question is, why after broadcasting, the size of the result y is [50,1] rather than [50,2], I think it should be [50,2], am I correct?
The # is not broadcasting but multiplication.
In python 3.5, the # operator was introduced for matrix
multiplication, following PEP465. This is implemented e.g. in numpy
as the matmul operator.
So the size of y is fine.
multiply a matrix of size [50,2] with a vector of size [2,1] will output a vector of size [50,1].
An example showing it more clearly is:
import torch
xx = torch.ones(3, 2)
zz = torch.tensor([[-1.0], [2.0]])
yy = xx # zz
print(xx)
print(zz)
print(yy)
# tensor([[1., 1.],
# [1., 1.],
# [1., 1.]])
# tensor([[-1.],
# [ 2.]])
# tensor([[1.],
# [1.],
# [1.]])
As you can see the third output is indeed just the multiplication of the 2 tensors.
If you wish to do broadcasting I recommend that you will refer to https://medium.com/ai%C2%B3-theory-practice-business/understanding-broadcasting-in-pytorch-ca9e9533f05f
In R, when I execute the code below:
> X=matrix(1,2,3)
> c=c(1,2,3)
> X*c
R gives out the following output:
[,1] [,2] [,3]
[1,] 1 3 2
[2,] 2 1 3
But when I do the below on Python:
>>> import numpy as np
>>> X=np.array([[1,1,1],[1,1,1]])
>>> c=np.array([1,2,3])
>>> X*c
the Python code above gives the following output:
array([[1, 2, 3],
[1, 2, 3]])
Is there any way that I can make the Python to come up with the identical output as R? I think I somehow have to tell Python that I want the numpy to multiply each element of the matrix X by each element of the vector c along the column, instead of along the row, but I am not sure how to go about this.
In [18]: np.reshape([1,2,3]*2,(2,3),order='F')
Out[18]:
array([[1, 3, 2],
[2, 1, 3]])
This starts with a list multiply, which is replication:
In [19]: [1,2,3]*2
Out[19]: [1, 2, 3, 1, 2, 3]
The rest uses numpy to reshape it into a (2,3) array, but with consecutive values going down, 'F' order.
Not knowning R, and in particular the c(1,2,3) expression, I can't say that's what's going on in R.
===
You talk about rows with columns, but I don't see how that works in your example. That said, we can easily perform outer like products
===
This reproduces your R_Product (at least in a few test cases):
In [138]: def foo(X,c):
...: X1 = X.ravel()
...: Y = np.resize(c,X1.shape)*X1
...: return Y.reshape(X.shape, order='F')
...:
In [139]: foo(np.ones((2,3)),np.arange(1,4))
Out[139]:
array([[1., 3., 2.],
[2., 1., 3.]])
In [140]: foo(np.arange(6).reshape(2,3),np.arange(1,4))
Out[140]:
array([[ 0, 6, 8],
[ 2, 3, 15]])
I'm using the resize function to replicate c to match the total number of elements of X. And order F to stack them in the desired column order. The default for numpy is order C.
In numpy replicating an array to match another is not common, at least not in this sense. Replicating by row or column, as in broadcasting is common. And of course reshaping.
I am the OP.
I was looking for a quick and easy solution, but I guess there is no straightforward functionality in Python that allows us to do this. So, I had to make a function that multiplies a matrix with a vector in the same manner that R does:
def R_product(X,c):
"""
Computes the regular R product
(not same as the matrix product) between
a 2D Numpy Array X, and a numpy vector c.
Args:
X: 2D Numpy Array
c: A Numpy vector
Returns: the output of X*c in R.
(This is different than X/*/c in R)
"""
X_nrow = X.shape[0]
X_ncol = X.shape[1]
X_dummy = np.zeros(shape=((X_nrow * X_ncol),1))
nrow = X_dummy.shape[0]
nc = nrow // len(c)
Y = np.zeros(shape=(nrow,1))
for j in range(X_ncol):
for u in range(X_nrow):
X_element = X[u,j]
if u == X_nrow - 1:
idx = X_nrow * (j+1) - 1
else:
idx = X_nrow * j + (u+1) - 1
X_dummy[idx,0] = X_element
for i in range(nc):
for j in range(len(c)):
Y[(i*len(c)+j):(i*len(c)+j+1),:] = (X_dummy[(i*len(c)+j):(i*len(c)+j+1),:]) * c[j]
for z in range(nrow-nc*len(c)):
Y[(nc*len(c)+z):(nc*len(c)+z+1),:] = (X_dummy[(nc*len(c)+z):(nc*len(c)+z+1),:]) * c[z]
return Y.reshape(X_ncol, X_nrow).transpose() # the answer I am looking for
Should work.
I have two numpy arrays of different shapes, but with the same length (leading dimension). I want to shuffle each of them, such that corresponding elements continue to correspond -- i.e. shuffle them in unison with respect to their leading indices.
This code works, and illustrates my goals:
def shuffle_in_unison(a, b):
assert len(a) == len(b)
shuffled_a = numpy.empty(a.shape, dtype=a.dtype)
shuffled_b = numpy.empty(b.shape, dtype=b.dtype)
permutation = numpy.random.permutation(len(a))
for old_index, new_index in enumerate(permutation):
shuffled_a[new_index] = a[old_index]
shuffled_b[new_index] = b[old_index]
return shuffled_a, shuffled_b
For example:
>>> a = numpy.asarray([[1, 1], [2, 2], [3, 3]])
>>> b = numpy.asarray([1, 2, 3])
>>> shuffle_in_unison(a, b)
(array([[2, 2],
[1, 1],
[3, 3]]), array([2, 1, 3]))
However, this feels clunky, inefficient, and slow, and it requires making a copy of the arrays -- I'd rather shuffle them in-place, since they'll be quite large.
Is there a better way to go about this? Faster execution and lower memory usage are my primary goals, but elegant code would be nice, too.
One other thought I had was this:
def shuffle_in_unison_scary(a, b):
rng_state = numpy.random.get_state()
numpy.random.shuffle(a)
numpy.random.set_state(rng_state)
numpy.random.shuffle(b)
This works...but it's a little scary, as I see little guarantee it'll continue to work -- it doesn't look like the sort of thing that's guaranteed to survive across numpy version, for example.
Your can use NumPy's array indexing:
def unison_shuffled_copies(a, b):
assert len(a) == len(b)
p = numpy.random.permutation(len(a))
return a[p], b[p]
This will result in creation of separate unison-shuffled arrays.
X = np.array([[1., 0.], [2., 1.], [0., 0.]])
y = np.array([0, 1, 2])
from sklearn.utils import shuffle
X, y = shuffle(X, y, random_state=0)
To learn more, see http://scikit-learn.org/stable/modules/generated/sklearn.utils.shuffle.html
Your "scary" solution does not appear scary to me. Calling shuffle() for two sequences of the same length results in the same number of calls to the random number generator, and these are the only "random" elements in the shuffle algorithm. By resetting the state, you ensure that the calls to the random number generator will give the same results in the second call to shuffle(), so the whole algorithm will generate the same permutation.
If you don't like this, a different solution would be to store your data in one array instead of two right from the beginning, and create two views into this single array simulating the two arrays you have now. You can use the single array for shuffling and the views for all other purposes.
Example: Let's assume the arrays a and b look like this:
a = numpy.array([[[ 0., 1., 2.],
[ 3., 4., 5.]],
[[ 6., 7., 8.],
[ 9., 10., 11.]],
[[ 12., 13., 14.],
[ 15., 16., 17.]]])
b = numpy.array([[ 0., 1.],
[ 2., 3.],
[ 4., 5.]])
We can now construct a single array containing all the data:
c = numpy.c_[a.reshape(len(a), -1), b.reshape(len(b), -1)]
# array([[ 0., 1., 2., 3., 4., 5., 0., 1.],
# [ 6., 7., 8., 9., 10., 11., 2., 3.],
# [ 12., 13., 14., 15., 16., 17., 4., 5.]])
Now we create views simulating the original a and b:
a2 = c[:, :a.size//len(a)].reshape(a.shape)
b2 = c[:, a.size//len(a):].reshape(b.shape)
The data of a2 and b2 is shared with c. To shuffle both arrays simultaneously, use numpy.random.shuffle(c).
In production code, you would of course try to avoid creating the original a and b at all and right away create c, a2 and b2.
This solution could be adapted to the case that a and b have different dtypes.
Very simple solution:
randomize = np.arange(len(x))
np.random.shuffle(randomize)
x = x[randomize]
y = y[randomize]
the two arrays x,y are now both randomly shuffled in the same way
James wrote in 2015 an sklearn solution which is helpful. But he added a random state variable, which is not needed. In the below code, the random state from numpy is automatically assumed.
X = np.array([[1., 0.], [2., 1.], [0., 0.]])
y = np.array([0, 1, 2])
from sklearn.utils import shuffle
X, y = shuffle(X, y)
from np.random import permutation
from sklearn.datasets import load_iris
iris = load_iris()
X = iris.data #numpy array
y = iris.target #numpy array
# Data is currently unshuffled; we should shuffle
# each X[i] with its corresponding y[i]
perm = permutation(len(X))
X = X[perm]
y = y[perm]
Shuffle any number of arrays together, in-place, using only NumPy.
import numpy as np
def shuffle_arrays(arrays, set_seed=-1):
"""Shuffles arrays in-place, in the same order, along axis=0
Parameters:
-----------
arrays : List of NumPy arrays.
set_seed : Seed value if int >= 0, else seed is random.
"""
assert all(len(arr) == len(arrays[0]) for arr in arrays)
seed = np.random.randint(0, 2**(32 - 1) - 1) if set_seed < 0 else set_seed
for arr in arrays:
rstate = np.random.RandomState(seed)
rstate.shuffle(arr)
And can be used like this
a = np.array([1, 2, 3, 4, 5])
b = np.array([10,20,30,40,50])
c = np.array([[1,10,11], [2,20,22], [3,30,33], [4,40,44], [5,50,55]])
shuffle_arrays([a, b, c])
A few things to note:
The assert ensures that all input arrays have the same length along
their first dimension.
Arrays shuffled in-place by their first dimension - nothing returned.
Random seed within positive int32 range.
If a repeatable shuffle is needed, seed value can be set.
After the shuffle, the data can be split using np.split or referenced using slices - depending on the application.
you can make an array like:
s = np.arange(0, len(a), 1)
then shuffle it:
np.random.shuffle(s)
now use this s as argument of your arrays. same shuffled arguments return same shuffled vectors.
x_data = x_data[s]
x_label = x_label[s]
There is a well-known function that can handle this:
from sklearn.model_selection import train_test_split
X, _, Y, _ = train_test_split(X,Y, test_size=0.0)
Just setting test_size to 0 will avoid splitting and give you shuffled data.
Though it is usually used to split train and test data, it does shuffle them too.
From documentation
Split arrays or matrices into random train and test subsets
Quick utility that wraps input validation and
next(ShuffleSplit().split(X, y)) and application to input data into a
single call for splitting (and optionally subsampling) data in a
oneliner.
This seems like a very simple solution:
import numpy as np
def shuffle_in_unison(a,b):
assert len(a)==len(b)
c = np.arange(len(a))
np.random.shuffle(c)
return a[c],b[c]
a = np.asarray([[1, 1], [2, 2], [3, 3]])
b = np.asarray([11, 22, 33])
shuffle_in_unison(a,b)
Out[94]:
(array([[3, 3],
[2, 2],
[1, 1]]),
array([33, 22, 11]))
One way in which in-place shuffling can be done for connected lists is using a seed (it could be random) and using numpy.random.shuffle to do the shuffling.
# Set seed to a random number if you want the shuffling to be non-deterministic.
def shuffle(a, b, seed):
np.random.seed(seed)
np.random.shuffle(a)
np.random.seed(seed)
np.random.shuffle(b)
That's it. This will shuffle both a and b in the exact same way. This is also done in-place which is always a plus.
EDIT, don't use np.random.seed() use np.random.RandomState instead
def shuffle(a, b, seed):
rand_state = np.random.RandomState(seed)
rand_state.shuffle(a)
rand_state.seed(seed)
rand_state.shuffle(b)
When calling it just pass in any seed to feed the random state:
a = [1,2,3,4]
b = [11, 22, 33, 44]
shuffle(a, b, 12345)
Output:
>>> a
[1, 4, 2, 3]
>>> b
[11, 44, 22, 33]
Edit: Fixed code to re-seed the random state
Say we have two arrays: a and b.
a = np.array([[1,2,3],[4,5,6],[7,8,9]])
b = np.array([[9,1,1],[6,6,6],[4,2,0]])
We can first obtain row indices by permutating first dimension
indices = np.random.permutation(a.shape[0])
[1 2 0]
Then use advanced indexing.
Here we are using the same indices to shuffle both arrays in unison.
a_shuffled = a[indices[:,np.newaxis], np.arange(a.shape[1])]
b_shuffled = b[indices[:,np.newaxis], np.arange(b.shape[1])]
This is equivalent to
np.take(a, indices, axis=0)
[[4 5 6]
[7 8 9]
[1 2 3]]
np.take(b, indices, axis=0)
[[6 6 6]
[4 2 0]
[9 1 1]]
If you want to avoid copying arrays, then I would suggest that instead of generating a permutation list, you go through every element in the array, and randomly swap it to another position in the array
for old_index in len(a):
new_index = numpy.random.randint(old_index+1)
a[old_index], a[new_index] = a[new_index], a[old_index]
b[old_index], b[new_index] = b[new_index], b[old_index]
This implements the Knuth-Fisher-Yates shuffle algorithm.
Shortest and easiest way in my opinion, use seed:
random.seed(seed)
random.shuffle(x_data)
# reset the same seed to get the identical random sequence and shuffle the y
random.seed(seed)
random.shuffle(y_data)
most solutions above work, however if you have column vectors you have to transpose them first. here is an example
def shuffle(self) -> None:
"""
Shuffles X and Y
"""
x = self.X.T
y = self.Y.T
p = np.random.permutation(len(x))
self.X = x[p].T
self.Y = y[p].T
With an example, this is what I'm doing:
combo = []
for i in range(60000):
combo.append((images[i], labels[i]))
shuffle(combo)
im = []
lab = []
for c in combo:
im.append(c[0])
lab.append(c[1])
images = np.asarray(im)
labels = np.asarray(lab)
I extended python's random.shuffle() to take a second arg:
def shuffle_together(x, y):
assert len(x) == len(y)
for i in reversed(xrange(1, len(x))):
# pick an element in x[:i+1] with which to exchange x[i]
j = int(random.random() * (i+1))
x[i], x[j] = x[j], x[i]
y[i], y[j] = y[j], y[i]
That way I can be sure that the shuffling happens in-place, and the function is not all too long or complicated.
Just use numpy...
First merge the two input arrays 1D array is labels(y) and 2D array is data(x) and shuffle them with NumPy shuffle method. Finally split them and return.
import numpy as np
def shuffle_2d(a, b):
rows= a.shape[0]
if b.shape != (rows,1):
b = b.reshape((rows,1))
S = np.hstack((b,a))
np.random.shuffle(S)
b, a = S[:,0], S[:,1:]
return a,b
features, samples = 2, 5
x, y = np.random.random((samples, features)), np.arange(samples)
x, y = shuffle_2d(train, test)
m,n=input()
a=[0]*n
for i in range(0,m):
a[i]=[0]*m
for i in range(0,m):
for j in range(0,n):
a[i][j]=input()
print a
Consider the above piece of code written in Python 2.7.4 to accept a 2-D array and then print it. This code functions well but it should accept any 2-D array means for example the values of m and could be 3,2 respectively let's say but it only accepts a square matrix. We cannot create a rectangular matrix because it gives the error: index out of range if the values of m and n are not equal. Is there any way to create a rectangular matrix just like we can do in C/C++ easily?
Numpy is a great module for fast linear algebra operations. You can create a rectangular array with Numpy which is essentially a matrix. (Numpy also has matrix functions too but they are a little more tedious to work with).
As an example, create a 3x4 array as follows
import numpy as np
input = np.zeros((3, 4)) #This creates a 3x4 array. It is good practice to initialize your array with 0's
input[0][3] = 5 #Fill out your array. The 0,0 index is the top left corner
In [42]:input
Out[42]:
array([[ 0., 0., 0., 5.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.]])
Proceed to fill out the rest of your rectangular matrix as per normal.
You can do something like this:
rows = input()
cols = input()
arr = [[None] * cols for _ in range(rows)]
for i in range(rows):
for j in range(cols):
print(arr)
arr[i][j] = input()
print(arr)
You should see this similar question: How to initialize a two-dimensional array in Python?
As far as getting your exact code to work, you just have m and n switched on lines 2 & 4:
m = input() # rows
n = input() # cols
a = [0] * m # Initialize rows
for i in range(0, m): # For each row (must be n rows)
a[i] = [0] * n # Initialize a column
for i in range(0, m): # For each row
for j in range(0, n): # For each column
a[i][j] = input() # Input a value
print a
I have a list of tuples e.g. like this:
l=[ (2,2,1), (2,4,0), (2,8,0),
(4,2,0), (4,4,1), (4,8,0),
(8,2,0), (8,4,0), (8,8,1) ]
and want to transform it to an numpy array like this (only z values in the matrix, corresponding to the sequence of x, y coordinates, the coordinates should be stored separately) ):
array([[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]])
I'm posting my solution below, but it's pretty low-level and I think there should be some higher-lever solution for this, either using matplotlib or numpy. Any idea?
One needs this kind of conversion to provide the arrays to matplotlib plotting functions like pcolor, imshow, contour.
It looks like np.unique with the return_inverse option fits the bill. For example,
In [203]: l[:,0]
Out[203]: array([2, 2, 2, 4, 4, 4, 8, 8, 8])
In [204]: np.unique(l[:,0], return_inverse = True)
Out[204]: (array([2, 4, 8]), array([0, 0, 0, 1, 1, 1, 2, 2, 2]))
np.unique returns a 2-tuple. The first array in the 2-tuple is an array of all the unique values in l[:,0]. The second array is the
index values associating values in array([2, 4, 8]) with values in the original array l[:,0]. It also happens to be the rank, since np.unique returns the unique values in sorted order.
import numpy as np
import matplotlib.pyplot as plt
l = np.array([ (2,2,1), (2,4,0), (2,8,0),
(4,2,0), (4,4,1), (4,8,0),
(8,2,0), (8,4,0), (8,8,1) ])
x, xrank = np.unique(l[:,0], return_inverse = True)
y, yrank = np.unique(l[:,1], return_inverse = True)
a = np.zeros((max(xrank)+1, max(yrank)+1))
a[xrank,yrank] = l[:,2]
fig = plt.figure()
ax = plt.subplot(111)
ax.pcolor(x, y, a)
plt.show()
yields
My solution first ranks the x and y values, and then creates the array.
l=[ (2,2,1), (2,4,0), (2,8,0),
(4,2,0), (4,4,1), (4,8,0),
(8,2,0), (8,4,0), (8,8,1) ]
def rankdata_ignoretied(data):
"""ranks data counting all tied values as one"""
# first translate the data values to integeres in increasing order
counter=0
encountered=dict()
for e in sorted(data):
if e not in encountered:
encountered[e]=counter
counter+=1
# then map the original sequence of the data values
result=[encountered[e] for e in data]
return result
x=[e[0] for e in l]
y=[e[1] for e in l]
z=[e[2] for e in l]
xrank=rankdata_ignoretied(x)
yrank=rankdata_ignoretied(y)
import numpy
a=numpy.zeros((max(xrank)+1, max(yrank)+1))
for i in range(len(l)):
a[xrank[i],yrank[i]]=l[i][2]
To use the resulting array for plotting one also needs the original x and y values, e.g.:
ax=plt.subplot(511)
ax.pcolor(sorted(set(x)), sorted(set(y)), a)
Anyone has a better idea of how to achieve this?
I don't understand why you're making this so complex. You can do it simply with:
array([
[cell[2] for cell in row] for row in zip(*[iter(x)] * 3)
])
Or perhaps more readably:
array([
[a[2], b[2], c[2]] for a, b, c in zip(x[0::3], x[1::3], x[2::3])
])
a solution using standard python construct set, list and sorted. if you don't have a lot of pointsit gains in readability even if slower than the numpy solution given by unutbu
l=[ (2,2,1), (2,4,0), (2,8,0),
(4,2,0), (4,4,1), (4,8,0),
(8,2,0), (8,4,0), (8,8,1) ]
#get the ranks of the values for x and y
xi = sorted(list(set( i[0] for i in l )))
yi = sorted(list(set( i[1] for i in l )))
a = np.zeros((len(xi),len(yi)))
#fill the matrix using the list.index
for x,y,v in l:
a[xi.index(x),yi.index(y)]=v
ax=plt.subplot(111)
ax.pcolor(array(xi), array(yi), a)