Develop Reference

No conversion path for dtype ('<U1') issue

I have a 2d list (Data_set) that contain a 3d array and a label(0 or 1), I want to make the h5py file with two datasets one for 3d array and the other for the label, this is my code for doing that:
`
data = []
label = []
for i in range(len(Data_set)):
data.append(Data_set[i][0])# 3d array
label.append(Data_set[i][1])#label
data = np.array(data)
label = np.array(label)
dt = np.dtype('int16')
with h5py.File(output_path+'dataset.h5', 'w') as hf:
hf.create_dataset('data',dtype=dt ,data=data, compression='lzf')
hf.create_dataset('label', dtype=dt, data=label, compression='lzf')
`
the content of the 2d list is shown in the image below:
but when I run the code it gives me an error: see the image below
please help me to solve the problem?

Your labels are not integers, they are strings, that's a problem for HDF5. Your error message relates to an array consisting of strings of length 1. See Strings in HDF5 for more details.
You can convert to integers before or after you construct your NumPy array, here are a couple of examples:
label = np.array(label).astype(int)
# or, label = np.array(list(map(int, label)))
Alternatively, since your values are 0 or 1, choosing bool may be more efficient:
label = np.array(label).astype(int).astype(bool)
Also, consider holding meta-data as attributes.

Writing a 3D Numpy array to a CSV file

I have a 3D Numpy array with the shape [1953,949,13]. I want to write it to a CSV file where each row should contain a 2D array of shape [949 13] and csv file should contain 1953 rows. I tried np.savetext and it supports only 1D and 2D arrays. Then I tried line by line writing to a CSV but it requires each matrix to be converted to a string. How can I get this done in python? My requirement is different from the question Storing values in a 3D array to csv

I'm not sure if it's the best way to doing it, but I faced the same problem and here's how I solved it.
import csv
import numpy as np
fil_name = 'file'
example = np.zeros((2,3,4))
example = example.tolist()
with open(fil_name+'.csv', 'w', newline='') as csvfile:
writer = csv.writer(csvfile, delimiter=',')
writer.writerows(example)
#to read file you saved
with open(fil_name+'.csv', 'r') as f:
reader = csv.reader(f)
examples = list(reader)
print(examples)
nwexamples = []
for row in examples:
nwrow = []
for r in row:
nwrow.append(eval(r))
nwexamples.append(nwrow)
print(nwexamples)

I used this method instead, not aware of any better method:
# reshaping the array from 3D matrice to 2D matrice.
arrReshaped = arr.reshape(arr.shape[0], -1)
# saving reshaped array to file.
np.savetxt(filename, arrReshaped)
# retrieving data from file.
loadedArr = np.loadtxt(filename)
# This loadedArr is a 2D array, therefore we need to convert it to the original array shape.
# reshaping to get original matrice with original shape.
loadedOriginal = loadedArr.reshape(loadedArr.shape[0], loadedArr.shape[1] // arr.shape[2], arr.shape[2])

How do I subset a 2D grid from another 2D grid in python?

I have gridded data over the contiguous United States and I'm trying to select a chunk of it over a specific area.
import numpy as np
from netCDF4 import Dataset
import matplotlib.pyplot as plt
filename = '/Users/me/myfile.nc'
full_data = Dataset(filename,'r')
latitudes = full_data.variables['latitude'][0,:,:]
longitudes = full_data.variables['longitude'][0,:,:]
temperature = full_data.variables['temperature'][0,:,:]
All three variables are 2-dimensional matrices of shape (337,451). I'm trying to do the following to get a sub-selection of the data over a specific region.
index = (latitudes>=44.0)&(latitudes<=45.0)&(longitudes>=-91.0)&(longitudes<=-89.0)
temp_subset = temperature[index]
lat_subset = latitudes[index]
lon_subset = longitudes[index]
I would expect all three of these variables to be 2-dimensional, but instead they all return a flattened array with a shape of (102,). I've tried another approach:
index2 = np.where((latitudes>=44.0)&(latitudes<=45.0)&(longitudes>=-91.0)&(longitudes<=-89.0))
temp = temperatures[index2[0],:]
temp2 = temp[:,index2[1]]
plt.imshow(temp2,origin='lower')
plt.colobar()
But my data looks quite incorrect. Is there a better way to get a 2D subset grid from a larger grid?

Edub,
I suggest looking on at numpy's matrix indexing documentation, specifically http://docs.scipy.org/doc/numpy-1.10.1/user/basics.indexing.html#other-indexing-options . Currently, you are providing two dimensions for indexing, but no slicing information (resulting in only receiving one dimensional results). I hope this proves useful!

Load sequence of PNGs into vtkImageData for 3D volume render using python

I have a sequence of about 100 PNG files containing 512x512 pre-segmented CAT scan data. I want to use vtk on Python to create a 3D model using marching cubes algorithm. The part that I don't know how to do is to load the sequence of PNG files and convert them to a single vtk pixel data object suitable for sending to the vtkDiscreteMarchingCubes algorithm.
I also think that I need to convert the pixel values of the PNG data because right now the data is in the alpha channel, so this needs to be converted into scalar data with values of zero and 1.

use vtkPNGreader and load in individual slices and then populate a vtkImageData which you can define the dimensions as and for each z-slice or image fill the image data form the output of the reader into your vtkImageData.
Rough pseudocode - not checked for bugs :)
import vtk
from vtk.util import numpy_support
pngfiles = glob.glob('*.png')
png_reader = vtk.vtkPNGReader()
png_reader.SetFileName(pngfiles[0])
x,y = png_reader.GetOutput().GetDimensions()
data_3d = np.zeros([x,y,len(pngfiles)])
for i,p in enumerate(png):
png_reader.SetFileName(pngfiles[0])
png_reader.Update()
img_data = png_reader.GetOutput()
data_3D[:,:,i] = numpy_support.vtk_to_numpy(img_data)
#save your 3D numpy array out.
data_3Dvtk = numpy_support.numpy_to_vtk(data_3D)

Just in case anyone stumbles on here looking for another way to do this only using vtk, you can use vtkImageAppend class.
def ReadImages(files):
reader = vtk.vtkPNGReader()
image3D = vtk.vtkImageAppend()
image3D.SetAppendAxis(2)
for f in files:
reader.SetFileName(f)
reader.Update()
t_img = vtk.vtkImageData()
t_img.DeepCopy(reader.GetOutput())
image3D.AddInputData(t_img)
image3D.Update()
return image3D.GetOutput()
for converting the data you can take a look at what the output of t_img.GetPointData().GetArray('PNGImage') gives and see if it is the expected value.

store/load numpy array from binary files

I would like to store and load numpy arrays from binary files. For that purposes, I created two small functions. Each binary file should contain the dimensionality of the given matrix.
def saveArrayToFile(data, fileName):
with open(fileName, 'w') as file:
a = array.array('f')
nSamples, ndim = data.shape
a.extend([nSamples, ndim]) # write number of elements and dimensions
a.fromstring(data.tostring())
a.tofile(file)
def readArrayFromFile(fileName):
_featDesc = np.fromfile(fileName, 'f')
_ndesc = int(_featDesc[0])
_ndim = int(_featDesc[1])
_featDesc = _featDesc[2:]
_featDesc = _featDesc.reshape([_ndesc, _ndim])
return _featDesc, _ndesc, _ndim
An example on how to use the functions is:
myarr=np.array([[7, 4],[3, 9],[1, 3]])
saveArrayToFile(myarr,'myfile.txt')
_featDesc, _ndesc, _ndim = readArrayFromFile('myfile.txt')
However, an error message of 'ValueError: total size of new array must be unchanged' is shown. My arrays can be of size MxN and MxM. Any suggestions are more than welcomed.
I think the problem might be in the saveArrayToFile function.
Best wishes,
Javier

Use numpy.save (and numpy.load) to dump (retrieve) numpy arrays to (from) a binary file.