I would like to apply a function to each of the 3x3 matrices in my (6890,6890,3,3) numpy array. Until now, I have tried using vectorization on a smaller example and with a simpler function which didn't work out.
def myfunc(x):
return np.linalg.norm(x)
m = np.arange(45).reshape(5,3,3)
t = m.shape[0]
r = np.zeros((t, t))
q = m[:,None,...] # m.swapaxes(1,2) # m[i] # m[j].T
f = np.vectorize(q, otypes=[np.float])
res = myfunc(f)
Is vectorization even the right approach to solve this problem efficiently or should I try something else? I've also looked into numpy.apply_along_axis but this only applies to 1D-subarrays.
You need loop over each element and apply function:
import numpy as np
# setup function
def myfunc(x):
return np.linalg.norm(x*2)
# setup data array
data = np.arange(45).reshape(5, 3, 3)
# loop over elements and update
for item in np.nditer(data, op_flags = ['readwrite']):
item[...] = myfunc(item)
If you need apply function for entire 3x3 array then use:
out_data = []
for item in data:
out_data.append(myfunc(item))
Output:
[14.2828568570857, 39.761790704142086, 66.4529909033446, 93.32202312423365, 120.24974012445931]
Related
import numpy as np
import pandas as pd
import cmath
a = np.array([[complex(3,6),complex(7,9),complex(2,8),complex(6,5)],
[complex(3,7),complex(7,9),complex(2,8),complex(6,5)],
[complex(3,8),complex(7,9),complex(2,8),complex(6,5)],
[complex(3,9),complex(7,9),complex(2,8),complex(6,5)],
[complex(3,1),complex(7,9),complex(2,8),complex(6,5)],
[complex(3,2),complex(7,9),complex(2,8),complex(6,5)],
[complex(3,3),complex(7,9),complex(2,8),complex(6,5)],
[complex(3,4),complex(7,9),complex(2,8),complex(6,5)],
])
l = np.array(['eval1_real','eval2_real','eval3_real','eval4_real','eval1_imag','eval2_imag','eval3_imag','eval4_imag'])
x = 1
for i in range(0, len(a),1):
w = a[i]
e1r = w[0].real
e1c = w[0].imag
e2r = w[1].real
e2c = w[1].imag
e3r = w[2].real
e3c = w[2].imag
e4r = w[3].real
e4c = w[3].imag
p = np.array([e1r, e1c, e2r, e2c, e3r, e3c, e4r, e4c])
m = np.insert(l,x,p,0)
x = x + 1
I tried for loop to separate but i cannot get those number to form together to become a full matrix
Is there a way to separate it altogether without using a loop or some array function i can put those together?
You should learn to use numpy-builtin functions for elemental operations on all elements. You can try,
result = np.dstack(
np.apply_along_axis(
lambda x: [x.real, x.imag], 0, a)
).flatten().reshape(8,8)
numpy.apply_along_axis
numpy.dstack
I have a function, for simplicity I'll call it shuffler and it takes an list, gives random a seed 17 and then prints that list shuffled.
def shuffler( n ):
import random
random.seed( 17 )
print( random.shuffle( n ) )
How would I create another function called unshuffler that "unshuffles" that list that is returned by shuffler(), bringing it back to the list I inputted into shuffler() assuming that I know the seed?
Just wanted to contribute an answer that's more compatible with functional patterns commonly used with numpy. Ultimately this solution should perform the fastest as it will take advantage of numpy's internal optimizations, which themselves can be further optimized via the use of projects like numba. It ought to be much faster than using conventional loop structures in python.
import numpy as np
original_data = np.array([23, 44, 55, 19, 500, 201]) # Some random numbers to represent the original data to be shuffled
data_length = original_data.shape[0]
# Here we create an array of shuffled indices
shuf_order = np.arange(data_length)
np.random.shuffle(shuf_order)
shuffled_data = original_data[shuf_order] # Shuffle the original data
# Create an inverse of the shuffled index array (to reverse the shuffling operation, or to "unshuffle")
unshuf_order = np.zeros_like(shuf_order)
unshuf_order[shuf_order] = np.arange(data_length)
unshuffled_data = shuffled_data[unshuf_order] # Unshuffle the shuffled data
print(f"original_data: {original_data}")
print(f"shuffled_data: {shuffled_data}")
print(f"unshuffled_data: {unshuffled_data}")
assert np.all(np.equal(unshuffled_data, original_data))
Here are two functions that do what you need:
import random
import numpy as np
def shuffle_forward(l):
order = range(len(l)); random.shuffle(order)
return list(np.array(l)[order]), order
def shuffle_backward(l, order):
l_out = [0] * len(l)
for i, j in enumerate(order):
l_out[j] = l[i]
return l_out
Example
l = range(10000); random.shuffle(l)
l_shuf, order = shuffle_forward(l)
l_unshuffled = shuffle_backward(l_shuf, order)
print l == l_unshuffled
#True
Reseed the random generator with the seed in question and then shuffle the list 1, 2, ..., n. This tells you exactly what ended up where in the shuffle.
In Python3:
import random
import numpy as np
def shuffle_forward(l):
order = list(range(len(l)); random.shuffle(order))
return list(np.array(l)[order]), order
def shuffle_backward(l, order):
l_out = [0] * len(l)
for i, j in enumerate(order):
l_out[j] = l[i]
return l_out
I have a function which gives me the index for a given value. Eg,
def F(value):
index = do_something(value)
return index
I want to use this index to fill a huge numpy array by 1s. Lets call array features
l = [1,4,2,3,7,5,3,6,.....]
NOTE: features.shape[0] = len(l)
for i in range(features.shape[0]):
idx = F(l[i])
features[i, idx] = 1
Is there a pythonic way to perform this (as the loop takes a lot of time if the array is huge)?
If you can vectorize F(value) you could write something like
indices = np.arange(features.shape[0])
feature_indices = F(l)
features.flat[indices, feature_indices] = 1
try this:
i = np.arange(features.shape[0]) # rows
j = np.vectorize(F)(np.array(l)) # columns
features[i,j] = 1
We have some arrays of the same shape and want to merge them.
By "merge", I mean output a new array with the sum of each i,j in each of the arrays in each position.
import numpy as np
first = np.array([[1,1],[1,1]])
second = np.array([[2,2],[2,2]])
third = np.array([[3,3],[3,3]])
The result should be:
[[6,6],
[6,6]]
Here's my code...but is there a cleaner way? I can't seem to find a built-in method:
def merge_arrays(arrays):
output = arrays[0]
for a in arrays[1:]:
for i,row in enumerate(a):
for j,col in enumerate(row):
output[i,j] += a[i,j]
return output
merge_arrays([first, second, third])
It's just output = first + second + third or np.sum([first, second, third], axis=0).
I'm rather new to NumPy. Anyone have an idea for making this code, especially the nested loops, more compact/efficient? BTW, dist and data are three-dimensional numpy arrays.
def interpolate_to_distance(self,distance):
interpolated_data=np.ndarray(self.dist.shape[1:])
for j in range(interpolated_data.shape[1]):
for i in range(interpolated_data.shape[0]):
interpolated_data[i,j]=np.interp(
distance,self.dist[:,i,j],self.data[:,i,j])
return(interpolated_data)
Thanks!
Alright, I'll take a swag with this:
def interpolate_to_distance(self, distance):
dshape = self.dist.shape
dist = self.dist.T.reshape(-1, dshape[-1])
data = self.data.T.reshape(-1, dshape[-1])
intdata = np.array([np.interp(distance, di, da)
for di, da in zip(dist, data)])
return intdata.reshape(dshape[0:2]).T
It at least removes one loop (and those nested indices), but it's not much faster than the original, ~20% faster according to %timeit in IPython. On the other hand, there's a lot of (probably unnecessary, ultimately) transposing and reshaping going on.
For the record, I wrapped it up in a dummy class and filled some 3 x 3 x 3 arrays with random numbers to test:
import numpy as np
class TestClass(object):
def interpolate_to_distance(self, distance):
dshape = self.dist.shape
dist = self.dist.T.reshape(-1, dshape[-1])
data = self.data.T.reshape(-1, dshape[-1])
intdata = np.array([np.interp(distance, di, da)
for di, da in zip(dist, data)])
return intdata.reshape(dshape[0:2]).T
def interpolate_to_distance_old(self, distance):
interpolated_data=np.ndarray(self.dist.shape[1:])
for j in range(interpolated_data.shape[1]):
for i in range(interpolated_data.shape[0]):
interpolated_data[i,j]=np.interp(
distance,self.dist[:,i,j],self.data[:,i,j])
return(interpolated_data)
if __name__ == '__main__':
testobj = TestClass()
testobj.dist = np.random.randn(3, 3, 3)
testobj.data = np.random.randn(3, 3, 3)
distance = 0
print 'Old:\n', testobj.interpolate_to_distance_old(distance)
print 'New:\n', testobj.interpolate_to_distance(distance)
Which prints (for my particular set of randoms):
Old:
[[-0.59557042 -0.42706077 0.94629049]
[ 0.55509032 -0.67808257 -0.74214045]
[ 1.03779189 -1.17605275 0.00317679]]
New:
[[-0.59557042 -0.42706077 0.94629049]
[ 0.55509032 -0.67808257 -0.74214045]
[ 1.03779189 -1.17605275 0.00317679]]
I also tried np.vectorize(np.interp) but couldn't get that to work. I suspect that would be much faster if it did work.
I couldn't get np.fromfunction to work either, as it passed (2) 3 x 3 (in this case) arrays of indices to np.interp, the same arrays you get from np.mgrid.
One other note: according the the docs for np.interp,
np.interp does not check that the x-coordinate sequence xp is increasing. If
xp is not increasing, the results are nonsense. A simple check for
increasingness is::
np.all(np.diff(xp) > 0)
Obviously, my random numbers violate the 'always increasing' rule, but you'll have to be more careful.