Develop Reference

Saving 3 dimensional array to a file in Python

I want to save a 3 dimensional arrays values to a txt or csv file in python.
dCx, dCy
I used:
numpy.savetxt('C:/Users/musa/Desktop/LOCO_All_tests/FODO_Example/AllQ/dCx.csv',dCx,delimiter=',')
numpy.savetxt('C:/Users/musa/Desktop/LOCO_All_tests/FODO_Example/AllQ/dCy.csv',dCy,delimiter=',')
And to load it again:
dCx = numpy.genfromtxt('C:/Users/musa/Desktop/LOCO_All_tests/FODO_Example/AllQ/dCx.csv', delimiter=',')
dCy = numpy.genfromtxt('C:/Users/musa/Desktop/LOCO_All_tests/FODO_Example/AllQ/dCy.csv', delimiter=',')
But i got the error massage
"Expected 1D or 2D array, got 3D array instead"
Si i wanted to change the 3d arrays first to 2 arrays and then save it to the files, and when uploaded again i convert it back to 3d for example:
dCx2 = np.array(dCx).reshape(np.array(dCx).shape[0], -1)
dCy2 = np.array(dCy).reshape(np.array(dCy).shape[0], -1)
and after loaded to variable named dCx3 and dCy3 i used:
dCx = np.array(dCx3).reshape(
np.array(dCx3).shape[0], np.array(dCx3).shape[1] // np.array(dCx).shape[2], np.array(dCx).shape[2])
#dCy = np.array(dCy3).reshape(
# np.array(dCy3).shape[0], np.array(dCy3).shape[1] // np.array(dCy).shape[2], np.array(dCy).shape[2])
I am looking for a better method that i can used in the saving the 3d arrays to file, or a method to convert the 2d into 3d without having to measure the original arrays every time as it is used in this line:
np.array(dCy).shape[2], np.array(dCy).shape[2])

Use numpy.save(filepath, data) and data = numpy.load(filepath).
These are binary file formats, and generic for any type of NumPy data

Try tofile. it works for in my case. but array will write in 1D
import numpy as np
arr=np.arange(0,21).reshape(7,3)
arr.tofile('file.txt',sep=',')
arr2=np.fromfile('file.txt',sep=',')

How to export 3D array into a single row in excel using python

I am attempting to export a large array of 3D points into excel.
import numpy as np
import pandas as pd
d = np.asarray(data)
df = pd.Dataframe(d)
df.to_csv("C:/Users/Fred/Desktop/test.csv")
This exports the data into rows as below:
3.361490011 -27.39559937 -2.934410095
4.573401244 -26.45699201 -3.845634521
.....
Each line representing the x,y,z coordinates. However, for my analysis, I would like that the 2nd row is moved to columns beside the 1st row, and so on, so that all the coordinates for one shape are on the one row of the excel. I tried turning the data into a string but this returned the above too.
The reason is so I can add some population characteristics to the row for each 3d shape. Thanks for any help that anyone can give.

you can use x = df.to_numpy().flatten() to flatten your data and then save it to csv using np.savetxt.

Unmasking and handling satellite data in python

I was working with satellite data and by reading the variable it created a masked array by masking the fill values. But now I can't extract any values from that masked array.
How can I unmask the array
The code is mentioned below
import glob
import os
import netCDF4 as nc
import numpy as np
#listing all nc files in this folder
os.chdir(r'I:\Data\AOD\MODIS_AQUA_AOD\2004')
myfiles = glob.glob('*.nc')
ds=nc.Dataset(myfiles[0])
#now myfiles has every nc file name saved as an array
#reading lat and lon one time because same band in every file
lats=ds.variables['lat'][:]
lons=ds.variables['lon'][:]
aods=[] #will save the AOD data in this array
# Creating Loop and reading AOD Data
for i in range(len(myfiles)):
dt=nc.Dataset(myfiles[0],'r')
aod=dt.variables['MYD08_D3_6_1_AOD_550_Dark_Target_Deep_Blue_Combined_Mean'][:]
aods.append(aod)
now I want to extract values of specific latitude and Longitude from the list of masked array list (aods) but I am unable to do so
any solutions?

Writing a 3D Numpy array to a CSV file

I have a 3D Numpy array with the shape [1953,949,13]. I want to write it to a CSV file where each row should contain a 2D array of shape [949 13] and csv file should contain 1953 rows. I tried np.savetext and it supports only 1D and 2D arrays. Then I tried line by line writing to a CSV but it requires each matrix to be converted to a string. How can I get this done in python? My requirement is different from the question Storing values in a 3D array to csv

I'm not sure if it's the best way to doing it, but I faced the same problem and here's how I solved it.
import csv
import numpy as np
fil_name = 'file'
example = np.zeros((2,3,4))
example = example.tolist()
with open(fil_name+'.csv', 'w', newline='') as csvfile:
writer = csv.writer(csvfile, delimiter=',')
writer.writerows(example)
#to read file you saved
with open(fil_name+'.csv', 'r') as f:
reader = csv.reader(f)
examples = list(reader)
print(examples)
nwexamples = []
for row in examples:
nwrow = []
for r in row:
nwrow.append(eval(r))
nwexamples.append(nwrow)
print(nwexamples)

I used this method instead, not aware of any better method:
# reshaping the array from 3D matrice to 2D matrice.
arrReshaped = arr.reshape(arr.shape[0], -1)
# saving reshaped array to file.
np.savetxt(filename, arrReshaped)
# retrieving data from file.
loadedArr = np.loadtxt(filename)
# This loadedArr is a 2D array, therefore we need to convert it to the original array shape.
# reshaping to get original matrice with original shape.
loadedOriginal = loadedArr.reshape(loadedArr.shape[0], loadedArr.shape[1] // arr.shape[2], arr.shape[2])

How can I save a matrix in a txt file and then open it again as a matrix in python?

I tried this:
import numpy as np
import os
outdir= "directory"
a = np.array([[1,2,3],[1,2,3]])
os.chdir(outdir)
np.savetxt("Image.bin", a)
f = open("directory/Image.bin")
m = np.fromfile(f, dtype=np.uint16)
print len(m)
ma = np.array(np.reshape(m, (2,3)))
print ma
But it returns this error message: "total size of new array must be unchanged"
I tried to change the dtype, but it dosen't work

You should use np.loadtxt: http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html
Try:
import numpy as np
a = np.array([[1,2,3],[1,2,3]])
np.savetxt("Image.bin", a)
m = np.loadtxt("Image.bin")
m now contains the same array as a.

numpy has built in functions for saving and loading arrays as binary files.
numpy.save('data.npy', data)
will create the file (it will append npy if you don't), and
data = numpy.load('data.npy')
will load it from the file. This is both faster and more space efficient than saving them as text files.