I have a 3D Numpy array with the shape [1953,949,13]. I want to write it to a CSV file where each row should contain a 2D array of shape [949 13] and csv file should contain 1953 rows. I tried np.savetext and it supports only 1D and 2D arrays. Then I tried line by line writing to a CSV but it requires each matrix to be converted to a string. How can I get this done in python? My requirement is different from the question Storing values in a 3D array to csv
I'm not sure if it's the best way to doing it, but I faced the same problem and here's how I solved it.
import csv
import numpy as np
fil_name = 'file'
example = np.zeros((2,3,4))
example = example.tolist()
with open(fil_name+'.csv', 'w', newline='') as csvfile:
writer = csv.writer(csvfile, delimiter=',')
writer.writerows(example)
#to read file you saved
with open(fil_name+'.csv', 'r') as f:
reader = csv.reader(f)
examples = list(reader)
print(examples)
nwexamples = []
for row in examples:
nwrow = []
for r in row:
nwrow.append(eval(r))
nwexamples.append(nwrow)
print(nwexamples)
I used this method instead, not aware of any better method:
# reshaping the array from 3D matrice to 2D matrice.
arrReshaped = arr.reshape(arr.shape[0], -1)
# saving reshaped array to file.
np.savetxt(filename, arrReshaped)
# retrieving data from file.
loadedArr = np.loadtxt(filename)
# This loadedArr is a 2D array, therefore we need to convert it to the original array shape.
# reshaping to get original matrice with original shape.
loadedOriginal = loadedArr.reshape(loadedArr.shape[0], loadedArr.shape[1] // arr.shape[2], arr.shape[2])
Related
I want to save a 3 dimensional arrays values to a txt or csv file in python.
dCx, dCy
I used:
numpy.savetxt('C:/Users/musa/Desktop/LOCO_All_tests/FODO_Example/AllQ/dCx.csv',dCx,delimiter=',')
numpy.savetxt('C:/Users/musa/Desktop/LOCO_All_tests/FODO_Example/AllQ/dCy.csv',dCy,delimiter=',')
And to load it again:
dCx = numpy.genfromtxt('C:/Users/musa/Desktop/LOCO_All_tests/FODO_Example/AllQ/dCx.csv', delimiter=',')
dCy = numpy.genfromtxt('C:/Users/musa/Desktop/LOCO_All_tests/FODO_Example/AllQ/dCy.csv', delimiter=',')
But i got the error massage
"Expected 1D or 2D array, got 3D array instead"
Si i wanted to change the 3d arrays first to 2 arrays and then save it to the files, and when uploaded again i convert it back to 3d for example:
dCx2 = np.array(dCx).reshape(np.array(dCx).shape[0], -1)
dCy2 = np.array(dCy).reshape(np.array(dCy).shape[0], -1)
and after loaded to variable named dCx3 and dCy3 i used:
dCx = np.array(dCx3).reshape(
np.array(dCx3).shape[0], np.array(dCx3).shape[1] // np.array(dCx).shape[2], np.array(dCx).shape[2])
#dCy = np.array(dCy3).reshape(
# np.array(dCy3).shape[0], np.array(dCy3).shape[1] // np.array(dCy).shape[2], np.array(dCy).shape[2])
I am looking for a better method that i can used in the saving the 3d arrays to file, or a method to convert the 2d into 3d without having to measure the original arrays every time as it is used in this line:
np.array(dCy).shape[2], np.array(dCy).shape[2])
Use numpy.save(filepath, data) and data = numpy.load(filepath).
These are binary file formats, and generic for any type of NumPy data
Try tofile. it works for in my case. but array will write in 1D
import numpy as np
arr=np.arange(0,21).reshape(7,3)
arr.tofile('file.txt',sep=',')
arr2=np.fromfile('file.txt',sep=',')
I have a .mat file, with only the following data inside :
cell{Name,Matrix(1610x10)} and I would like to obtain the matrix data in a numpy array to process it.
Thanks
You could save the data as CSV and read the data with python.
After this transform it to a numpy array.
http://www.mikesoltys.com/2014/04/06/readingwriting-to-csv-in-matlab/
in python:
csvData = []
with open("DATA.csv") as data:
csv_reader_objekt = csv.reader(data)
for row in csv_reader_objekt:
csvData.append(row)
DataArray = np.array(csvData)
I have a dataset with the following shape saved as a 3-dimensional array
(7352, 128, 6)
I want to save my data as 6 different files based on the 3rd dimension of the array
The code I used is below:
np.savetxt(filepath+'/'+dataName1+'.csv', normalizedX[:,:,0], delimiter=',')
np.savetxt(filepath+'/'+dataName2+'.csv', normalizedX[:,:,1], delimiter=',')
np.savetxt(filepath+'/'+dataName3+'.csv', normalizedX[:,:,2], delimiter=',')
np.savetxt(filepath+'/'+dataName4+'.csv', normalizedX[:,:,3], delimiter=',')
np.savetxt(filepath+'/'+dataName5+'.csv', normalizedX[:,:,4], delimiter=',')
np.savetxt(filepath+'/'+dataName6+'.csv', normalizedX[:,:,5], delimiter=',')
The shape of normalizedX[:,:,0] is (7352, 128) before saving them.
When I try to load the text with the following code:
def load_file(filepath):
dataframe = pd.read_csv(filepath, header=None, delim_whitespace=True)
return dataframe.values
I get a shape of (7352, 1), I lost data from my 2nd dimension!
What is the problem here?
You used , as a delimiter in np.savetxt so you should not put delim_whitespace='True' in your read_csv function
I have a 2d list (Data_set) that contain a 3d array and a label(0 or 1), I want to make the h5py file with two datasets one for 3d array and the other for the label, this is my code for doing that:
`
data = []
label = []
for i in range(len(Data_set)):
data.append(Data_set[i][0])# 3d array
label.append(Data_set[i][1])#label
data = np.array(data)
label = np.array(label)
dt = np.dtype('int16')
with h5py.File(output_path+'dataset.h5', 'w') as hf:
hf.create_dataset('data',dtype=dt ,data=data, compression='lzf')
hf.create_dataset('label', dtype=dt, data=label, compression='lzf')
`
the content of the 2d list is shown in the image below:
but when I run the code it gives me an error: see the image below
please help me to solve the problem?
Your labels are not integers, they are strings, that's a problem for HDF5. Your error message relates to an array consisting of strings of length 1. See Strings in HDF5 for more details.
You can convert to integers before or after you construct your NumPy array, here are a couple of examples:
label = np.array(label).astype(int)
# or, label = np.array(list(map(int, label)))
Alternatively, since your values are 0 or 1, choosing bool may be more efficient:
label = np.array(label).astype(int).astype(bool)
Also, consider holding meta-data as attributes.
I would like to store and load numpy arrays from binary files. For that purposes, I created two small functions. Each binary file should contain the dimensionality of the given matrix.
def saveArrayToFile(data, fileName):
with open(fileName, 'w') as file:
a = array.array('f')
nSamples, ndim = data.shape
a.extend([nSamples, ndim]) # write number of elements and dimensions
a.fromstring(data.tostring())
a.tofile(file)
def readArrayFromFile(fileName):
_featDesc = np.fromfile(fileName, 'f')
_ndesc = int(_featDesc[0])
_ndim = int(_featDesc[1])
_featDesc = _featDesc[2:]
_featDesc = _featDesc.reshape([_ndesc, _ndim])
return _featDesc, _ndesc, _ndim
An example on how to use the functions is:
myarr=np.array([[7, 4],[3, 9],[1, 3]])
saveArrayToFile(myarr,'myfile.txt')
_featDesc, _ndesc, _ndim = readArrayFromFile('myfile.txt')
However, an error message of 'ValueError: total size of new array must be unchanged' is shown. My arrays can be of size MxN and MxM. Any suggestions are more than welcomed.
I think the problem might be in the saveArrayToFile function.
Best wishes,
Javier
Use numpy.save (and numpy.load) to dump (retrieve) numpy arrays to (from) a binary file.