I have grid data at .csv format:
srcPath = "/data.csv"
f = open(srcPath,"r")
lines = f.readlines()
f.close()
miss = -9999.
ny,nx = 360,720
dlat = 0.5
dlon = 0.5
lat0 = -90.0
lon0 = -180.0
Lat = arange(-90, 90+0.001, dlat)
Lon = arange(-180, 180+0.001, dlon)
a2dat0 = ones([ny,nx],float32)*miss
a2dat1 = ones([ny,nx],float32)*miss
a2dat3 = ones([ny,nx],float32)*miss
a2dat4 = ones([ny,nx],float32)*miss
a2dat5 = ones([ny,nx],float32)*miss
a2dat6 = ones([ny,nx],float32)*miss
for line in lines:
lon, lat, v0,v1,v2,v3,v4,v5,v6 =map(float,line.split(","))
x = int(floor((lon - lon0)/dlon))
y = int(floor((lat - lat0)/dlat))
a2dat0 [y,x] = v0
a2dat1 [y,x] = v1
a2dat2 [y,x] = v2
a2dat3 [y,x] = v3
a2dat4 [y,x] = v4
a2dat5 [y,x] = v5
a2dat6 [y,x] = v6
X, Y = meshgrid(Lon, Lat)
m= Basemap(projection='robin',llcrnrlon=-180.,llcrnrlat=-90.,urcrnrlon=180.,urcrnrlat=90.,resolution='i',lon_0=0)
xi, yi = m(X, Y)
Now I want to make loof over "a2dat" like:
for i in range (0,7,1):
print a2dat+str(i)+":" ,a2dat+str(i).shape
or
for i in range (0,7,1):
cs+str(i) = m.pcolor(xi,yi,a2dat+str(i),cmap=cmap, norm=norm, )
to plot 7 different global maps using basemap and "a2dat" data, but It gives the back error, So how could I make the loop over these data name?
I think your dynamic names could be replaced with lists as follows (this isn't runnable code, so may have some bugs):
data = [np.ones([ny,nx], float32)*miss for _ in range(7)]
# data = [a2dat0, a2dat1, ... a2dat6] # or a list of the arrays
for line in lines:
lon, lat, *vs =map(float,line.split(",")) # py3 * unpacking
# vs = alist[2:] in py2?
x = int(floor((lon - lon0)/dlon))
y = int(floor((lat - lat0)/dlat))
for d,v in zip(data, vs):
d[y,x] = v
(Once the arrays are collecting a list, their names, if any, in the global environment aren't important.)
and
for i,d in enumerate(data):
print("a2dat%d: "%i ,data.shape)
csdata = [m.pcolor(xi, yi, d, ...) for d in data]
While it is possible to use for i in range(...): loops, generating the index with enumerate is more idiomatic. We're encouraged to think about iterating over a list of objects, not over the indices of those elements.
if you want to dynamically interpret python code from a string you need to use eval or exec. Also your print statement is wrong because it should not have :, and the first a2dat should be a string since you just want to get the text (also note I added a ,).
for i in range (0,7):
print "a2dat"+str(i)+":", eval("a2dat"+str(i)).shape
eval("a2dat"+str(i)) will get you the array object when it's evaluated. You can read up on the difference between eval and exec on your own.
Related
I have 3 numpy matrices:
One contains pixels positions in X (x_pos), another pixel positions in Y (y_pos) and a last one containing pixel values (p_value)
I would like to use these 3 matrices to build a results image
With loops I have this result:
#Resulting image
res = np.zeros((128,128,3), dtype = np.uint8)
for i in range(x_pos.shape[0]):
for j in range(x_pos.shape[1]):
# Get coordinates
x = x_pos[i][j]
y = y_pos[i][j]
res[y,x] = p_value[i][j]
With large matrices (2048*2048) this code already takes a lot of time. Is it possible to optimize this code without using a nested loop?
I specify that the positions in the pos_x and pos_y matrices do not necessarily follow each other, there may be holes or duplicate values
It should be possible using np.meshgrid
i = np.arange(0, x.shape[0])
j = np.arange(0, x.shape[1])
i_1, j_1 = np.meshgrid(i, j, indexing='ij')
res[y_1.ravel(),x_1.ravel()] = p_value[i_1.ravel(),j_1.ravel()]
First use consistent numpy 2d array indexing:
x = x_pos[i,j]
y = y_pos[i,j]
res[y,x] = p_value[i,j]
Now instead of scalar i,j use arrays
i = np.arange(n); j = np.arange(m)
You didn't provida [mcve] so I won't try to demonstrate that th
Thanks to #hpaulj and #ai2ys answer the problem is solved.
Here is a comparison of the results in terms of execution speed:
import numpy as np
import cv2
import time
m_size = 4096
m_x = np.random.randint(0,m_size,(m_size,m_size), dtype = np.uint16)
m_y = np.random.randint(0,m_size,(m_size,m_size), dtype = np.uint16)
p_value = np.ones((m_size,m_size), dtype = np.uint8)
#Meshgrid method:
out = np.zeros((m_size,m_size),dtype=np.uint8)
start = time.time()
i = np.arange(0, m_x.shape[0])
j = np.arange(0, m_x.shape[1])
i_1, j_1 = np.meshgrid(i, j, indexing='ij')
out[m_x.ravel(),m_y.ravel()] = p_value[i_1.ravel(),j_1.ravel()]
end = time.time()
print("Meshgrid: {} s".format(end - start))
#No for loop method:
out = np.zeros((m_size,m_size),dtype=np.uint8)
start = time.time()
i = np.arange(m_x.shape[0])
j = np.arange(m_y.shape[1])
x = m_x[i,j]
y = m_y[i,j]
out[x,y] = p_value[i,j]
end = time.time()
print("No loop: {} s".format(end - start))
#For loop method:
out = np.zeros((m_size,m_size),dtype=np.uint8)
start = time.time()
for i in range(m_x.shape[0]):
for j in range(m_y.shape[1]):
x = m_x[i,j]
y = m_y[i,j]
out[x,y] = p_value[i,j]
end = time.time()
print("Nested loop: {} s".format(end - start))
#Output:
Meshgrid: 0.4837045669555664 s
No loop: 0.3600656986236572 s
Nested loop: 13.10097336769104 s
I have set of lists (i.e. list x , list y, list z), e.g.
x = ['41.95915452', '41.96333025', '41.98135503', '41.95096716', '41.96504172', '41.96526867', '41.98068483', '41.98117072', '41.98059828', '41.95915452', '41.96333025', '41.98135503', '41.95096716']
y = ['12.60718918', '12.62725589', '12.6201431', '12.60017199', '12.62774075', '12.62800706', '12.62812394', '12.6278259', '12.62810614', '12.60718918', '12.62725589', '12.6201431', '12.60017199']
z = ['9.215398066', '8.249650758', '8.791595671', '8.246394455', '9.27132698', '5.667547722', '7.783268126', '9.471492129', '9.668210684', '9.215398066', '8.249650758', '8.791595671', '8.246394455']
There are such around 800 lists. I have to create a 3*3 matrix from the each of the lists x, y and z such that [x1, y1, z1], one of its row should be like ['41.95915452', '12.60718918', '9.215398066' ] and list must contain at least than 4 entries.
My code :
for i in np.arange(41.70, 42.10, 0.05):
#print(round(i,2), end=', ')
for j in np.arange(12.30, 12.80, 0.05):
# print(round(j,2), end=', ')
for k in np.arange(0,26,5):
#print("\n")
#print(round(i,2),round(j,2),k, end=', ')
xmax = round(i+0.05,2)
ymax = round(j+ 0.05,2)
zmax = round(k+5,2)
#print("Voxel",xmax,ymax,zmax)
v = []
x1 = []
y1 = []
z1 = []
count = 0;
with open('a.csv') as csvfile:
plots = csv.reader(csvfile,delimiter=',')
for rows in plots:
if(float(rows[0]) >= i and float(rows[0])<= xmax and float(rows[1]) >=j and float(rows[1])<=ymax and float(rows[2])>=k and float(rows[2])<=zmax):
#print("points", float(rows[0]),float(rows[1]),float(rows[2]))
x1.append(rows[0])
y1.append(rows[1])
z1.append(rows[2])
count= count+1
#f = open("demofile2.txt", "a")
#f.write(str(i)+","+str(j)+","+str(k)+","+str(count)+"\n")
#f.write(text)
#f.close()
#print(count)
if(count > 3):
v1 = [i,j,k]
v.append(v1)
print(v)
print(x1)
print(y1)
print(z1)
print("\n")
Use numpy vstack and transpose.
Try this code.
np.vstack([x, y, z]).T
If you want the output is list, then use
np.vstack([x, y, z]).T.tolist()
you can use np.c_ to concatenate along axis and slice the matrix.
res = np.c_[x,y,z][:3,:3]
output
array([['41.95915452', '12.60718918', '9.215398066'],
['41.96333025', '12.62725589', '8.249650758'],
['41.98135503', '12.6201431', '8.791595671']], dtype='<U11')
I have created a code that returns the output that I am after - 2 graphs with multiple lines on each graph. However, the code is slow and quite big (in terms of how many lines of code it takes). I am interested in any improvements I can make that will help me to get such graphs faster, and make my code more presentable.
Additionally, I would like to add more to my graphs (axis names and titles is what I am after). Normally, I would use plt.xlabel,plt.ylabel and plt.title to do so, however I couldn't quite understand how to use them here. The aim here is to add a line to each graph after each loop ( I have adapted this piece of code to do so).
I should note that I need to use Python for this task (so I cannot change to anything else) and I do need Sympy library to find values that are plotted in my graphs.
My code so far is as follows:
import matplotlib.pyplot as plt
import sympy as sym
import numpy as np
sym.init_printing()
x, y = sym.symbols('x, y') # defining our unknown probabilities
al = np.arange(20,1000,5).reshape((196,1)) # values of alpha/beta
prob_of_strA = []
prob_of_strB = []
colours=['r','g','b','k','y']
pen_values = [[0,-5,-10,-25,-50],[0,-25,-50,-125,-250]]
fig1, ax1 = plt.subplots()
fig2, ax2 = plt.subplots()
for j in range(0,len(pen_values[1])):
for i in range(0,len(al)): # choosing the value of beta
A = sym.Matrix([[10, 50], [int(al[i]), pen_values[0][j]]]) # defining matrix A
B = sym.Matrix([[pen_values[1][j], 50], [int(al[i]), 10]]) # defining matrix B
sigma_r = sym.Matrix([[x, 1-x]]) # defining the vector of probabilities
sigma_c = sym.Matrix([y, 1-y]) # defining the vector of probabilities
ts1 = A * sigma_c ; ts2 = sigma_r * B # defining our utilities
y_sol = sym.solvers.solve(ts1[0] - ts1[1],y,dict = True) # solving for y
x_sol = sym.solvers.solve(ts2[0] - ts2[1],x,dict = True) # solving for x
prob_of_strA.append(y_sol[0][y]) # adding the value of y to the vector
prob_of_strB.append(x_sol[0][x]) # adding the value of x to the vector
ax1.plot(al,prob_of_strA,colours[j],label = ["penalty = " + str(pen_values[0][j])]) # plotting value of y for a given penalty value
ax2.plot(al,prob_of_strB,colours[j],label = ["penalty = " + str(pen_values[1][j])]) # plotting value of x for a given penalty value
ax1.legend() # showing the legend
ax2.legend() # showing the legend
prob_of_strA = [] # emptying the vector for the next round
prob_of_strB = [] # emptying the vector for the next round
You can save a couple of lines by initializing your empty vectors inside the loop. You don't have to bother re-defining them at the end.
for j in range(0,len(pen_values[1])):
prob_of_strA = []
prob_of_strB = []
for i in range(0,len(al)): # choosing the value of beta
A = sym.Matrix([[10, 50], [int(al[i]), pen_values[0][j]]]) # defining matrix A
B = sym.Matrix([[pen_values[1][j], 50], [int(al[i]), 10]]) # defining matrix B
sigma_r = sym.Matrix([[x, 1-x]]) # defining the vector of probabilities
sigma_c = sym.Matrix([y, 1-y]) # defining the vector of probabilities
ts1 = A * sigma_c ; ts2 = sigma_r * B # defining our utilities
y_sol = sym.solvers.solve(ts1[0] - ts1[1],y,dict = True) # solving for y
x_sol = sym.solvers.solve(ts2[0] - ts2[1],x,dict = True) # solving for x
prob_of_strA.append(y_sol[0][y]) # adding the value of y to the vector
prob_of_strB.append(x_sol[0][x]) # adding the value of x to the vector
ax1.plot(al,prob_of_strA,colours[j],label = ["penalty = " + str(pen_values[0][j])]) # plotting value of y for a given penalty value
ax2.plot(al,prob_of_strB,colours[j],label = ["penalty = " + str(pen_values[1][j])]) # plotting value of x for a given penalty value
ax1.legend() # showing the legend
ax2.legend() # showing the legend
I have 5 different trajectories for my project. First I read them from the file, then save them to a list by parsing the file. After that I digitized these values. I want to align x and y coordinates separately and plot x and y coordinates together on a grid.
This is what I have done so far. I use dtw package in python, but it takes two lists and gives the path as an array. How can I convert it to an aligned trajectory of 5 different lists?
x_dict = {}
y_dict = {}
for x in ["1", "2", "3", "4", "5"]:
file = open("data-" + x + ".txt", encoding="latin-1")
data = file.read()
pos_list = re.findall(r'position:(.*?)orientation:', data, re.DOTALL)
or_list = re.findall(r'orientation:(.*?)scale:', data, re.DOTALL)
for i in range(len(pos_list)):
pos_list[i] = pos_list[i].replace('\n','')
regexx = re.compile(r'x: (.*?) y:')
regexy = re.compile(r'y: (.*?) z:')
posx_list = [m.group(1) for l in pos_list for m in [regexx.search(l)] if m]
posx_list = list(map(float, posx_list))
posy_list = [m.group(1) for l in pos_list for m in [regexy.search(l)] if m]
posy_list = list(map(float, posy_list))
bins = numpy.linspace(-1, 1, 100)
digitized_x = numpy.digitize(posx_list, bins)
digitized_y = numpy.digitize(posy_list, bins)
x_dict[x] = digitized_x
y_dict[x] = digitized_y
dist, cost, acc, path = dtw(y_dict["5"], y_dict["4"], dist= euclidean)
plt.imshow(acc.T, origin='lower', cmap=cm.gray, interpolation='nearest')
plt.plot(path[0], path[1], 'w')
plt.xlim((-0.5, acc.shape[0]-0.5))
plt.ylim((-0.5, acc.shape[1]-0.5))
My question still related to read many files at once. I tried to read data that consist of many variables and then plot it in one figure. Can anybody help me to make this code much simpler by using loop ?
base_dir = 'C:/INTERMITTENT/MARCH/'
nc_file1 = '20080305.faieb3p4e.nc'
nc_file2 = '20080306.faieb3p4e.nc'
# Change directory
os.chdir(base_dir)
# Open file
fd1 = nc.Dataset(nc_file1, 'r')
fd2 = nc.Dataset(nc_file2, 'r')
# Read variables
beam1 = fd1.variables['beam'][:]
rng1 = fd1.variables['range'][:]
tim1 = fd1.variables['time'][:]
pwr1 = fd1.variables['pwr'][:]
nfft1 = fd1.variables['nfft'][0]
pn1 = fd1.variables['pnoise'][:]
beam2 = fd2.variables['beam'][:]
rng2 = fd2.variables['range'][:]
tim2 = fd2.variables['time'][:]
pwr2 = fd2.variables['pwr'][:]
nfft2 = fd2.variables['nfft'][0]
pn2 = fd2.variables['pnoise'][:]
# Close netCDF file
fd1.close()
fd2.close()
# Specify beam
ibeam1 = 0
ibeam2 = 0
# Time convertion
tim1 = tim1/3600.0
tim2 = tim2/3600.0
#Plot
p_plot1 = pwr1[ibeam1]
for it1 in range(len(tim1)):
p_plot1[it1] = p_plot1[it1] - pn1[ibeam1][it1] - 10.*np.log10(nfft1)
p_plot1 = p_plot1.transpose()
p_plot2 = pwr2[ibeam2]
for it2 in range(len(tim2)):
p_plot2[it2] = p_plot2[it2] - pn2[ibeam2][it2] - 10.*np.log10(nfft2)
p_plot2 = p_plot2.transpose()
#Plot colorbar
fig.colorbar(plot1, ax=axs[0])
fig.colorbar(plot2, ax=axs[1])
# Set X and Y axis lower/uppser limit
set_xy = range(4)
set_xy[0] = 18.0 # x min
set_xy[1] = 30.0 # x max
set_xy[2] = 90.0 # y min
set_xy[3] = 170.0 # y max
pl.axis(set_xy)
# Set labels
pl.xlabel('time (hours)')
pl.ylabel('range (km)')
pl.show()
if I have more than 2 files, let's say I have 30 files, how do I use loop in this case.
Thank you in advance.
Use a list of file names and iterate:
nc_files = ['20080305.faieb3p4e.nc', '20080306.faieb3p4e.nc']
for fname in nc_files:
fd = nc.Dataset(fname, 'r')
... # do stuff
Note that you would then store your results in lists rather than naming them beam1, beam2, etc.
Edit: Here is a rewrite of your code. I noticed that your data was only used to create a plot and then not referred to again, so I rewrite the values during each loop. Also note that I used enumerate while iterating over your file names to get a counter variable i:
base_dir = 'C:/INTERMITTENT/MARCH/'
nc_fnames = ['20080305.faieb3p4e.nc', '20080306.faieb3p4e.nc']
# beams
ibeams = [0, 0]
# Change directory
os.chdir(base_dir)
for i, fname in enumerate(nc_fnames):
# Open file
fd = nc.Dataset(fname, 'r')
# Read variables
beam = fd.variables['beam'][:]
rng = fd.variables['range'][:]
tim = fd.variables['time'][:]
pwr = fd.variables['pwr'][:]
nfft = fd.variables['nfft'][0]
pn = fd.variables['pnoise'][:]
# Close netCDF file
fd.close()
# Specify beam
ibeam = ibeams[i]
# Time convertion
tim = tim/3600.0
#Plot
p_plot = pwr[ibeam]
for it in range(len(tim)):
p_plot[it] = p_plot[it] - pn[ibeam][it] - 10.*np.log10(nfft)
p_plot = p_plot.transpose()
#Plot colorbar
fig.colorbar(p_plot, ax=axs[i])
# Set X and Y axis lower/uppser limit
set_xy = range(4)
set_xy[0] = 18.0 # x min
set_xy[1] = 30.0 # x max
set_xy[2] = 90.0 # y min
set_xy[3] = 170.0 # y max
pl.axis(set_xy)
# Set labels
pl.xlabel('time (hours)')
pl.ylabel('range (km)')
pl.show()