Develop Reference

What is view method doing on numpy matrix that changes the shape?

So I have a function:
def create_vecs(colnames):
return np.matrix(data[colnames]).view(dtype=np.float64).reshape(-1, 3)
when I apply this function on my data, first part gets the columns of interest and returns a numpy matrix of size 1340*3. but then I'm not sure what view is doing on my data that it doesn't let my data to be reshaped to three columns. I'm confused how this view method works and how to change it so that I can reshape my data back to three columns.

When you say:
.reshape(-1, 3)
python reshape so that the second index be 3 unit and first index adapts to whatever it gets
for example 1340*3 you have 4020 unit data
so if you use
.reshape(-1, 5, 4)
the shape of the matrix becomes
(201, 5, 4)
I hope i was clear.

How to add (not append) a 2d array to each 2d array of a 3d array?

I want to vectorize the process of adding a 2d array to every 2d array inside a 3d array.
I imported an image file using image from matplotlib
data = image.imread('test.jpg')
Then I tried to add the average of each RGB array to another array of the same shape as data
data2 = np.zeros_like(data)
data3 = np.average(data, axis=2)
for i in range(len(data2[0,0,:])):
data2[:,:,i] = data3
I just want to vectorize the above 2 line code to one line

Convert data3 to the result datatype and then broadcast/repeat after extending to 3D with np.newaxis/None -
b = data3.astype(data.dtype)
data2_out = np.broadcast_to(b[...,None], data.shape)
The output would simply be a view into b and hence we are gaining memory-efficiency there.
If you need an output with its own memory space, we can force the copy with data2_out.copy() or use np.repeat, like so -
np.repeat(b[...,None],data.shape[2],axis=2)
If you already have the output array data2 initialized and just want to assign into it, we can do so with extending data3 to 3D and this might more intuitive in some scenarios too, like so -
data2[:] = data3[...,None]

How to add several vectors to numpy structered array and call matrix later from fieldname?

Hey guys Ii need help..
I want to use tensorflows data import, where data is loaded by calling the features/labels vectors from a structured numpy array.
https://www.tensorflow.org/programmers_guide/datasets#consuming_numpy_arrays
I want to create such an structured array by adding consecutively the 2 vectors (feature_vec and label_vec) to an numpy structured array.
import numpy as np
# example vectors
feature_vec= np.arange(10)
label_vec = np.arange(10)
# structured array which should get the vectors
struc_array = np.array([feature_vec,label_vec],dtype=([('features',np.float32), ('labels',np.float32)]))
# How can I add now new vectors to struc_array?
struc_array.append(---)
I want later when this array is loaded from file call either the feature vectors (which is a matrix now) by using the fieldname:
with np.load("/var/data/training_data.npy") as data:
features = data["features"] # matrix containing feature vectors as rows
labels = data["labels"] #matrix containing labels vectors as rows
Everything I tried to code was complete crap.. never got a correct output..
Thanks for your help!

Don't create a NumPy array and then append to it. That doesn't really make sense, as NumPy arrays have a fixed size and require a full copy to append a single row or column. Instead, create a list, append to it, then construct the array at the end:
vecs = [feature_vec,label_vec]
dtype = [('features',np.float32), ('labels',np.float32)]
# append as many times as you want:
vecs.append(other_vec)
dtype.append(('other', np.float32))
struc_array = np.array(vecs, dtype=dtype)
Of course, you probably need ot

Unfortunately, this doesn't solve the problem.
i want to get just the labels or the features from structured array by using:
labels = struc_array['labels']
features = struc_array['features']
But when i use the structured array like you did, labels and also features contains all given appended vectors:
import numpy as np
feature_vec= np.arange(10)
label_vec = np.arange(0,5,0.5)
vecs = [feature_vec,label_vec]
dtype = [('features',np.float32), ('labels',np.float32)]
other_vec = np.arange(6,11,0.5)
vecs.append(other_vec)
dtype.append(('other', np.float32))
struc_array = np.array(vecs, dtype=dtype)
# This contains all vectors.. not just the labels vector
labels = struc_array['labels']
# This also contains all vectors.. not just the feature vector
features = struc_array['features']

Put multiple 2d numpy arrays into 3d numpy array

I'm trying to put multiple 2-D numpy arrays into one 3-D numpy array and then save the 3-D numpy array as a compressed file to a directory for later use.
I have a list that I'm looping through which will compute forecasts for different hazards. A forecast for each hazard (a 129x185 numpy array) will be computed one at a time. I want to then put each forecast array into an empty 129x185x7 numpy array.
hazlist = ['allsvr', 'torn', 'sigtorn', 'hail', 'sighail', 'wind', 'sigwind']
# Create 3-D empty numpy array
grid = np.zeros(shape=(129,185,7))
for i,haz in enumerate(hazlist):
*do some computation to create forecast array for current hazard*
# Now have 2-D 129x185 forecast array
print fcst
# Place 2-D array into empty 3-D array.
*Not sure how to do this...*
# Save 3-D array to .npz file in directory when all 7 hazard forecasts are done.
np.savez_compressed('pathtodir/3dnumpyarray.npz')
But, I want to give each forecast array it's own grid name inside the 3-D array so that if I want a certain one (like tornadoes) I can just call it with:
filename = np.load('pathtodir/3dnumpyarray.npz')
arr = filename['torn']
It would be greatly appreciated if someone were able to assist me. Thanks.

It sounds like you actually want to use a dictionary. Each dictionary entry could be a 2D array with the reference name as the key:
hazlist = ['allsvr', 'torn', 'sigtorn', 'hail', 'sighail', 'wind', 'sigwind']
# Create empty dictionary
grid = {}
for i,haz in enumerate(hazlist):
*do some computation to create forecast array for current hazard*
# Now have 2-D 129x185 forecast array
print fcst
# Place 2-D array into dictionary.
grid[haz] = fcst # Assuming fcst is the 2D array?
# Save 3-D array to npz file
np.savez_compressed("output", grid)
It might be best to save this as a JSON file. If the data needs to be compressed you can refer to this question and answer as to saving json in gzipped format, or this one may be clearer.
It's not clear from your example, but my assumption in the above code is that fcst is the 2D array that corresponds to the label haz in each iteration of the loop.

represent an index inside a list as x,y in python

I have a list which contains 1000 integers. The 1000 integers represent 20X50 elements of dimensional array which I read from a file into the list.
I need to walk through the list with an indicator in order to find close elements to each other. I want that my indicator will be represented not only by a simple index i, but as a two indices x,y so I can know where is my indicator along the list.
I tried to reshape the list like that:
data = np.array( l )
shape = ( 20, 50 )
data.reshape( shape )
but I don't know how to access the data array.
Update: Is there any way to find the indices of x, y for an integers that are smaller than NUM(let's say NUM=12)

According to documentation of numpy.reshape , it returns a new array object with the new shape specified by the parameters (given that, with the new shape, the amount of elements in the array remain unchanged) , without changing the shape of the original object, so when you are calling the data.reshape() function you should also assign it back to data for it to reflect in data.
Example -
data = data.reshape( shape ) # where shape = (20,50)
Also, another way to change the shape, is to directly assign the new shape to the data.shape property.
Example -
shape = (20,50)
data.shape = shape # where shape is the new shape