My question still related to read many files at once. I tried to read data that consist of many variables and then plot it in one figure. Can anybody help me to make this code much simpler by using loop ?
base_dir = 'C:/INTERMITTENT/MARCH/'
nc_file1 = '20080305.faieb3p4e.nc'
nc_file2 = '20080306.faieb3p4e.nc'
# Change directory
os.chdir(base_dir)
# Open file
fd1 = nc.Dataset(nc_file1, 'r')
fd2 = nc.Dataset(nc_file2, 'r')
# Read variables
beam1 = fd1.variables['beam'][:]
rng1 = fd1.variables['range'][:]
tim1 = fd1.variables['time'][:]
pwr1 = fd1.variables['pwr'][:]
nfft1 = fd1.variables['nfft'][0]
pn1 = fd1.variables['pnoise'][:]
beam2 = fd2.variables['beam'][:]
rng2 = fd2.variables['range'][:]
tim2 = fd2.variables['time'][:]
pwr2 = fd2.variables['pwr'][:]
nfft2 = fd2.variables['nfft'][0]
pn2 = fd2.variables['pnoise'][:]
# Close netCDF file
fd1.close()
fd2.close()
# Specify beam
ibeam1 = 0
ibeam2 = 0
# Time convertion
tim1 = tim1/3600.0
tim2 = tim2/3600.0
#Plot
p_plot1 = pwr1[ibeam1]
for it1 in range(len(tim1)):
p_plot1[it1] = p_plot1[it1] - pn1[ibeam1][it1] - 10.*np.log10(nfft1)
p_plot1 = p_plot1.transpose()
p_plot2 = pwr2[ibeam2]
for it2 in range(len(tim2)):
p_plot2[it2] = p_plot2[it2] - pn2[ibeam2][it2] - 10.*np.log10(nfft2)
p_plot2 = p_plot2.transpose()
#Plot colorbar
fig.colorbar(plot1, ax=axs[0])
fig.colorbar(plot2, ax=axs[1])
# Set X and Y axis lower/uppser limit
set_xy = range(4)
set_xy[0] = 18.0 # x min
set_xy[1] = 30.0 # x max
set_xy[2] = 90.0 # y min
set_xy[3] = 170.0 # y max
pl.axis(set_xy)
# Set labels
pl.xlabel('time (hours)')
pl.ylabel('range (km)')
pl.show()
if I have more than 2 files, let's say I have 30 files, how do I use loop in this case.
Thank you in advance.
Use a list of file names and iterate:
nc_files = ['20080305.faieb3p4e.nc', '20080306.faieb3p4e.nc']
for fname in nc_files:
fd = nc.Dataset(fname, 'r')
... # do stuff
Note that you would then store your results in lists rather than naming them beam1, beam2, etc.
Edit: Here is a rewrite of your code. I noticed that your data was only used to create a plot and then not referred to again, so I rewrite the values during each loop. Also note that I used enumerate while iterating over your file names to get a counter variable i:
base_dir = 'C:/INTERMITTENT/MARCH/'
nc_fnames = ['20080305.faieb3p4e.nc', '20080306.faieb3p4e.nc']
# beams
ibeams = [0, 0]
# Change directory
os.chdir(base_dir)
for i, fname in enumerate(nc_fnames):
# Open file
fd = nc.Dataset(fname, 'r')
# Read variables
beam = fd.variables['beam'][:]
rng = fd.variables['range'][:]
tim = fd.variables['time'][:]
pwr = fd.variables['pwr'][:]
nfft = fd.variables['nfft'][0]
pn = fd.variables['pnoise'][:]
# Close netCDF file
fd.close()
# Specify beam
ibeam = ibeams[i]
# Time convertion
tim = tim/3600.0
#Plot
p_plot = pwr[ibeam]
for it in range(len(tim)):
p_plot[it] = p_plot[it] - pn[ibeam][it] - 10.*np.log10(nfft)
p_plot = p_plot.transpose()
#Plot colorbar
fig.colorbar(p_plot, ax=axs[i])
# Set X and Y axis lower/uppser limit
set_xy = range(4)
set_xy[0] = 18.0 # x min
set_xy[1] = 30.0 # x max
set_xy[2] = 90.0 # y min
set_xy[3] = 170.0 # y max
pl.axis(set_xy)
# Set labels
pl.xlabel('time (hours)')
pl.ylabel('range (km)')
pl.show()
Related
So I'm running some measurements using a keithley 2450 source meter with this code:
def res_trace(n = None, max_v = None, min_v = None, data_points = None, r_crit = None,
ilim = None):
beep(164,0.5)
# check values and provide defaults
n = n or 0
data_points = data_points or 100
max_v = max_v or 0.5
min_v = min_v or -0.5
r_crit = r_crit or 1e+7
ilim = ilim or 'MAXimum'
v = np.linspace( min_v, max_v, num = data_points)
i = []
res_run = []
# reset keithley
# just so we can use them without any prior settings
reset()
# set up I measurement systems
keith.write(':SENSe:FUNCtion "CURR"')
keith.write(':SENSe:CURRent:RANGe:AUTO 1')
keith.write(':SENSe:CURRent:UNIT AMP')
keith.write(':SENSe:CURRent:NPLCycles DEFault')
keith.write(':SENSe:COUNt DEFault')
keith.write(':SENSe:CURRent:OCOM ON')
# set up V source, (Hi Michale here!)
keith.write(':SOURce:FUNCtion VOLT')
keith.write(':SOURce:VOLTage:RANGe '+str(max_v))
keith.write(':SOURce:VOLTage:ILIMit '+ ilim)
# Turn keith on
keith_output('on')
for j in v:
keith.write(':SOURce:VOLT '+str(j))
itemp = float(keith.query(':MEASure:CURRent?'))
i.append(itemp)
###
# turn them off
keith_output('off')
# plot
plt.figure()
plt.title('Res trace # '+str(n))
plt.plot(v,i, c = 'm')
plt.xlabel('V')
plt.ylabel('I')
plt.grid()
plt.show()
I'm currently running the script and it takes a few seconds for keithley to take measurements and return values. I'd like to get a way to live plot the data within each loop as its being collected, but I have no idea how to go about this and want the process to be as simple as possible. Any suggestions?
Thanks
You can use Jupyterplot and create a realtime plot like this:
from jupyterplot import ProgressPlot
import numpy as np
pp = ProgressPlot()
for i in range(1000):
pp.update(np.sin(i / 100))
pp.finalize()
Above is an image of the contour plot. I am having difficulties with seeing the yellow color, so I was trying to change the colors. I am not sure how to change the colors. In addition, I was wondering how to change the position of the labels on the lines. Below is the code I used to create the graph. The code will extract values from a txt file that is in the following format
1.0;3.0;0.11793
1.0;23.0;0.1441
1.0;43.0;0.14525
1.0;63.0;0.14554
1.0;83.0;0.14541
1.0;103.0;0.14533
1.0;123.0;0.14533
There is a lot more data than this in the text file it extract from, but this just show an example whats in the txt file.
import numpy as np
import matplotlib.pyplot as plt
from random import randint
#----- EXTRACTS AND PROCESS DATA FROM .TXT FILE ------------------
data_file = open('UC4results.txt','r') #opens file
lines = data_file.readlines() #reads file
data_file.close()
r_list = [] #creates list
h_list = []
keff_list = []
for i in range(len(lines)):
lines[i] = lines[i].strip('\n')
line_list = lines[i].split(';')
r_list.append(float(line_list[0])) #grabs values in the first column
h_list.append(float(line_list[1])) #grabs values in the second column
keff_list.append(float(line_list[2])) #grabs values in the third column
R,H = np.meshgrid(np.unique(r_list),np.unique(h_list))
R = 2*np.array(R)
h_diam =[]
for i in range(len(R)):
placeholder = []
for diam in range(len(R[0])):
placeholder.append(H[i][diam]/(R[i][diam]))
h_diam.append(placeholder)
keff_array = []
keff_storage = []
count = 0
for i in range(len(keff_list)):
keff_storage.append(keff_list[i])
count += 1
if count % len(np.unique(r_list)) == 0: # adds the list after its the appropriate length, and starts a new one
keff_array.append(keff_storage)
keff_storage = []
plt.figure(figsize=(20,20), dpi=400)
contour_graph = plt.contour(R,H,keff_array,[0.800,0.900,1.000,1.100,1.200]) #plots contour plot
plt.xlabel('Diameter (cm)', size = 30)
plt.ylabel('Height (cm)', size = 30)
plt.clabel(contour_graph, fontsize=20)
plt.xticks(size = 30)
plt.yticks(size = 30)
plt.xlim(0,60)
plt.ylim(0,60)
plt.grid()
I have grid data at .csv format:
srcPath = "/data.csv"
f = open(srcPath,"r")
lines = f.readlines()
f.close()
miss = -9999.
ny,nx = 360,720
dlat = 0.5
dlon = 0.5
lat0 = -90.0
lon0 = -180.0
Lat = arange(-90, 90+0.001, dlat)
Lon = arange(-180, 180+0.001, dlon)
a2dat0 = ones([ny,nx],float32)*miss
a2dat1 = ones([ny,nx],float32)*miss
a2dat3 = ones([ny,nx],float32)*miss
a2dat4 = ones([ny,nx],float32)*miss
a2dat5 = ones([ny,nx],float32)*miss
a2dat6 = ones([ny,nx],float32)*miss
for line in lines:
lon, lat, v0,v1,v2,v3,v4,v5,v6 =map(float,line.split(","))
x = int(floor((lon - lon0)/dlon))
y = int(floor((lat - lat0)/dlat))
a2dat0 [y,x] = v0
a2dat1 [y,x] = v1
a2dat2 [y,x] = v2
a2dat3 [y,x] = v3
a2dat4 [y,x] = v4
a2dat5 [y,x] = v5
a2dat6 [y,x] = v6
X, Y = meshgrid(Lon, Lat)
m= Basemap(projection='robin',llcrnrlon=-180.,llcrnrlat=-90.,urcrnrlon=180.,urcrnrlat=90.,resolution='i',lon_0=0)
xi, yi = m(X, Y)
Now I want to make loof over "a2dat" like:
for i in range (0,7,1):
print a2dat+str(i)+":" ,a2dat+str(i).shape
or
for i in range (0,7,1):
cs+str(i) = m.pcolor(xi,yi,a2dat+str(i),cmap=cmap, norm=norm, )
to plot 7 different global maps using basemap and "a2dat" data, but It gives the back error, So how could I make the loop over these data name?
I think your dynamic names could be replaced with lists as follows (this isn't runnable code, so may have some bugs):
data = [np.ones([ny,nx], float32)*miss for _ in range(7)]
# data = [a2dat0, a2dat1, ... a2dat6] # or a list of the arrays
for line in lines:
lon, lat, *vs =map(float,line.split(",")) # py3 * unpacking
# vs = alist[2:] in py2?
x = int(floor((lon - lon0)/dlon))
y = int(floor((lat - lat0)/dlat))
for d,v in zip(data, vs):
d[y,x] = v
(Once the arrays are collecting a list, their names, if any, in the global environment aren't important.)
and
for i,d in enumerate(data):
print("a2dat%d: "%i ,data.shape)
csdata = [m.pcolor(xi, yi, d, ...) for d in data]
While it is possible to use for i in range(...): loops, generating the index with enumerate is more idiomatic. We're encouraged to think about iterating over a list of objects, not over the indices of those elements.
if you want to dynamically interpret python code from a string you need to use eval or exec. Also your print statement is wrong because it should not have :, and the first a2dat should be a string since you just want to get the text (also note I added a ,).
for i in range (0,7):
print "a2dat"+str(i)+":", eval("a2dat"+str(i)).shape
eval("a2dat"+str(i)) will get you the array object when it's evaluated. You can read up on the difference between eval and exec on your own.
I wrote a code a while ago that processes spectra using data from text files and performing calculations on them. I started with a code that just does everything line-by-line without any functions, and despite being long, it finishes running in 2.11 seconds (according to %%timeit). Below is that original code, labeled as such.
However, I wanted to put my code into functions instead, to allow for easier readability and usage with different models in the future. Even though I'm using all the same steps as I did before (but this time inside my functions), it is so much slower. This code is also below. Now, I have to wait for about 15-20 minutes to get the same outputs. Why is it so much slower, and is there any way I can make it significantly faster but still use functions?
Original Code:
import re
import matplotlib.pyplot as plt
import numpy as np
import scipy.interpolate
filename = 'bpass_spectra.txt'
extinctionfile = 'ExtinctionLawPoints.txt' # from R_V = 4.0
pointslist = []
datalist = []
speclist = []
# Constants
Msun = 1.98892e30 # solar mass [kg]
h = 4.1357e-15 # Planck's constant [eV s]
c = float(3e8) # speed of light [m/s]
# Read spectra file
f = open(filename, 'r')
rawspectra = f.readlines()
met = re.findall('Z\s=\s(\d*\.\d+)', rawspectra[0])
del rawspectra[0]
for i in range(len(rawspectra)):
newlist = rawspectra[i].split(' ')
datalist.append(newlist)
# Read extinction curve data file
rawpoints = open(extinctionfile, 'r').readlines()
for i in range(len(rawpoints)):
newlst = re.split('(?!\S)\s(?=\S)|(?!\S)\s+(?=\S)', rawpoints[i])
pointslist.append(newlst)
pointslist = pointslist[3:]
lambdalist = [float(item[0]) for item in pointslist]
k_abslist = [float(item[4]) for item in pointslist]
xvallist = [(c*h)/(lamb*1e-6) for lamb in lambdalist]
k_interp = scipy.interpolate.interp1d(xvallist, k_abslist)
# Create new lists
Elist = [float(item[0]) for item in datalist]
speclambdalist = [h*c*1e9/E for E in Elist]
z1list = [float(item[1]) for item in datalist]
speclist.extend(z1list)
met = met[0]
klist = [None]*len(speclist)
Loutlist = [None]*len(speclist)
Tlist = [None]*len(speclist)
# Define parameters
b = 2.0
R = 1.0
z = 1.0
Mgas = 1.0 # mass of gas, input
Mhalo = 2e41 # mass of dark matter halo, known
if float(met) > 0.0052:
DGRlist = [50.0*np.exp(-2.21)*float(met)]*len(speclist)
elif float(met) <= 0.0052:
DGRlist = [((50.0*float(met))**3.15)*np.exp(-0.96)]*len(speclist)
for i in range(len(speclist)):
if float(Elist[i]) <= 4.1357e-3: # frequencies <= 10^12 Hz
klist[i] = 0.1*(float(Elist[i])/(1000.0*h))**b # extinction law [cm^2/g]
elif float(Elist[i]) > 4.1357e-3: # frequencies > 10^12 Hz
klist[i] = k_interp(Elist[i]) # interpolated function's value at Elist[i]
Mdustlist = [Mgas*DGR for DGR in DGRlist] # dust mass
Rhalo = 0.784*(0.27**2.0)*(0.7**(-2.0/3.0))*float(10.0/(1.0+z))*((Mhalo/(1e8*Msun))**(1.0/3.0))
Rdust = 0.018*Rhalo # [kpc]
for i in range(len(speclist)):
Tlist[i] = 3*Mdustlist[i]*klist[i]/(4*np.pi*Rdust)
Linlist = [float(spectra)*R for spectra in speclist]
# Outgoing luminosity as function of wavelength
for i in range(len(Linlist)):
Loutlist[i] = Linlist[i]*np.exp(-Tlist[i])
# Test the calculation
print "LIN ELEMENTS 0 AND 1000:", Linlist[0], Linlist[1000]
print "LOUT ELEMENTS 0 AND 1000:", Loutlist[0], Loutlist[1000]
New "function-ized" Code (much slower):
import re
import matplotlib.pyplot as plt
import numpy as np
import scipy.interpolate
# Required files and lists
filename = 'bpass_spectra.txt' # number of columns = 4
extinctionfile = 'ExtinctionLawPoints.txt' # R_V = 4.0
datalist = []
if filename == 'bpass_spectra.txt':
filetype = 4
else:
filetype = 1
if extinctionfile == 'ExtinctionLawPoints.txt':
R_V = 4.0
else:
R_V = 1.0 #to be determined
# Constants
M_sun = 1.98892e30 # solar mass [kg]
h = 4.1357e-15 # Planck's constant [eV s]
c = float(3e8) # speed of light [m/s]
# Inputs
beta = 2.0
R = 1.0
z = 1.0
M_gas = 1.0
M_halo = 2e41
# Read spectra file
f = open(filename, 'r')
rawlines = f.readlines()
met = re.findall('Z\s=\s(\d*\.\d+)', rawlines[0])
del rawlines[0]
for i in range(len(rawlines)):
newlist = rawlines[i].split(' ')
datalist.append(newlist)
# Read extinction curve data file
rawpoints = open(extinctionfile, 'r').readlines()
def interpolate(R_V, rawpoints, Elist, j):
pointslist = []
if R_V == 4.0:
for i in range(len(rawpoints)):
newlst = re.split('(?!\S)\s(?=\S)|(?!\S)\s+(?=\S)', rawpoints[i])
pointslist.append(newlst)
pointslist = pointslist[3:]
lambdalist = [float(item[0]) for item in pointslist]
k_abslist = [float(item[4]) for item in pointslist]
xvallist = [(c*h)/(lamb*1e-6) for lamb in lambdalist]
k_interp = scipy.interpolate.interp1d(xvallist, k_abslist)
return k_interp(Elist[j])
# Dust extinction function
def dust(interpolate, filetype, datalist, beta, R, z, M_gas, M_halo, met):
speclist = []
if filetype == 4:
metallicity = float(met[0])
Elist = [float(item[0]) for item in datalist]
speclambdalist = [h*c*1e9/E for E in Elist]
met1list = [float(item[1]) for item in datalist]
speclist.extend(met1list)
klist, Tlist = [None]*len(speclist), [None]*len(speclist)
if metallicity > 0.0052:
DGRlist = [50.0*np.exp(-2.21)*metallicity]*len(speclist) # dust to gas ratio
elif metallicity <= 0.0052:
DGRlist = [((50.0*metallicity)**3.15)*np.exp(-0.96)]*len(speclist)
for i in range(len(speclist)):
if Elist[i] <= 4.1357e-3: # frequencies <= 10^12 Hz
klist[i] = 0.1*(float(Elist[i])/(1000.0*h))**beta # extinction law [cm^2/g]
elif Elist[i] > 4.1357e-3: # frequencies > 10^12 Hz
klist[i] = interpolate(R_V, rawpoints, Elist, i) # interpolated function's value at Elist[i]
Mdustlist = [M_gas*DGR for DGR in DGRlist] # dust mass
R_halo = 0.784*(0.27**2.0)*(0.7**(-2.0/3.0))*float(10/(1+z))*((M_halo/(1e8*M_sun))**(1.0/3.0))
R_dust = 0.018*R_halo # [kpc]
# Optical depth calculation
Tlist = [3*Mdustlist[i]*klist[i]/(4*np.pi*R_dust) for i in range(len(speclist))]
# Ingoing and outgoing luminosities as functions of wavelength
Linlist = [float(spectra)*R for spectra in speclist]
Loutlist = [Linlist[i]*np.exp(-Tlist[i]) for i in range(len(speclist))]
return speclambdalist, Linlist, Loutlist
print dust(interpolate, filetype, datalist, beta, R, z, M_gas, M_halo, met)
Even when I only have the function return Loutlist instead of the tuple of 3 lists, it's still extremely slow. Any ideas on why this is? Also, I'm going to want to return the tuple and then plot speclambdalist versus Linlist, and also plot speclambdalist versus Loutlist on the same plot. But I'm under the impression that each time I call dust(interpolate, filetype, datalist, beta, R, z, M_gas, M_halo, met)[i] where i = 0, 1, or 2 (I'll be doing this multiple times), it'll have to run the function again each time. Is there any way to bypass these extra runs to further increase speed? Thank you!
for line, images_files in zip(lines, image_list):
info = line.split(',')
image_index = [int(info[0])]
box_coordiante1 = [info[2]]
box_coordiante2 = [info[3]]
box_coordiante3 = [info[4]]
box_coordiante4 = [info[5]]
prev_image_num = 1
for image_number in image_index: #### read each other image_number
if prev_image_num != image_number: # if read 11111 but appear different number such as 2, 3 and ect
prev_image_num = image_number # the different number becomes pre_image_num(it was 1)
#box_coordinate = [] # empty box_coordinate
#box_coordinate.append(info[2:6])
#print box_coordinate
# box_coordinate.append() #insert 2 to 6 axis
rect = plt.Rectangle((int(box_coordiante1), int(box_coordiante2)), int(box_coordiante3), int(box_coordiante4), linewidth=1, edgecolor='r', facecolor='none')
ax.add_patch(rect)
im = cv2.imread(images_files)
im = im[:, :, (2, 1, 0)]
# # Display the image
plt.imshow(im)
plt.draw()
plt.pause(0.1)
plt.cla()
I am supposed to draw boxes on each picture.
For showing boxes on each picture,
i guess that gather location of boxes and show them at that same time.
So i used a way using LIST to plt.Rectanle
but it said "TypeError: int() argument must be a string or a number, not 'list'"
Are there other ways??
Umm, I just did just. I don't know if this is what you wanted though.
x = 10
y = 10
a = []
for unit for range(x):
a.append(0)
for unit for range(y):
print(a)
I'm not very familiar with Python, but it seems like you want a plain number in the variables image_index and box_coordinateN. It looks like you're assigning single-element arrays to them. Try changing:
image_index = [int(info[0])] // list containing one element: int(info[0])
box_coordiante1 = [info[2]]
box_coordiante2 = [info[3]]
box_coordiante3 = [info[4]]
box_coordiante4 = [info[5]]
to:
image_index = int(info[0]) // number: int(info[0])
box_coordiante1 = info[2]
box_coordiante2 = info[3]
box_coordiante3 = info[4]
box_coordiante4 = info[5]
The answer above is carelessly sloppy and incorrect Python.
It must be rewritten and corrected as follows:
x = 10
y = 10
a = []
for unit in range(x):
a.append(0)
for unit in range(y):
print(a)