Develop Reference

Plot 4D data heatmap in Python

hey how can I plot a 2D heatmap in 3D? Now I create a python script to make a 2D Heatmap Plot with data from CSV (CSV format: x,y,z,v).
For example:
First csv
0,000;-110,000;110,000;0,101
Second csv
0,000;-66,000;110,000;0,104
Third csv
0,000;-22,000;110,000;0,119
....
In this example, it is a heatmap in xz-plane and I create e.g. five more plots, so that I can insert six xz-plane Plots in a 3D room.
In 4D heatmap plot with matplotlib there is a very nice example for doing it. But I don't know how to use it in my case.
import numpy as np
import os
import matplotlib.pyplot as plt
from scipy.interpolate import griddata
'Create a list for every parameter'
x = []
y = []
z = []
v = []
file_path = "path/."
'Insert data from csv into lists'
for root, dirs, files in os.walk(file_path, topdown=False):
for name in files:
if name[-4:] != '.csv': continue
with open(os.path.join(root, name)) as data:
data = np.genfromtxt((line.replace(',', '.') for line in data), delimiter=";")
if data[1] == 22:
x.append(data[0])
y.append(data[1])
z.append(data[2])
v.append(data[3])
'Create axis data'
xi = np.linspace(min(x), max(x), 1000)
zi = np.linspace(min(z), max(z), 1000)
vi = griddata((x, z), v, (xi[None,:], zi[:,None]), method='cubic')
'Create the contour plot'
CS = plt.contourf(xi, zi, vi, 20, cmap=plt.cm.rainbow)
plt.title("Heatmap xz-plane", y=1.05,
fontweight="bold")
plt.xlabel("length x in cm")
plt.xticks(np.arange(0, 201, step=40))
plt.ylabel("height z in cm")
plt.yticks(np.arange(110, 251, step=20))
cbar = plt.colorbar()
cbar.set_label("velocity v in m/s", labelpad=10)
plt.savefig('testplot.png', dpi=400)
plt.show()
Satisfying the request of #keepAlive wishing to see the result of his untested answer... :
it actually works great :-)

Disclaimer: I am the author of the cited example, so I think that copying/pasting myself is not really a problem.
Note that your dataset does not look (at least) 3-dimensional. But I will assume there is an unwilling selection bias.
You first need to aggregate your "points" per level of altitude, which I assume is the third component of your vectors. They will be constitutive of your planes once gathered.
# libraries
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import scipy.interpolate as si
from matplotlib import cm
import collections as co # <------------------
import pandas as pd
import numpy as np
planes = co.defaultdict(list)
for root, dirs, files in os.walk(file_path, topdown=False):
# [...]
# [...]
# [...]
# [...]
# [...]
level = data[2] # <------ third component.
planes[level].append(data)
Now, at that stage, we have a list of arrays per level. Let's define our grids_maker function
def grids_maker(arrays_list, colnames=list('xyzg')):
# 0- The idea behind `list('xyzg')` is only to change the order
# of names, not the names as such. In case for example you
# want to use another component than the third to organize
# your planes.
# 1- Instantiate a dataframe so as to minimize the modification
# of the function copied/pasted pasted from
# https://stackoverflow.com/a/54075350/4194079
# 2- Pandas is also going to do some other jobs for us, such as
# stacking arrays, etc....
df = pd.DataFrame(arrays_list, columns=colnames)
# Make things more legible
xy = df.loc[:, ['x', 'y']]
x = xy.x
y = xy.y
z = df.z
g = df.g
reso_x = reso_y = 50
interp = 'cubic' # or 'nearest' or 'linear'
# Convert the 4d-space's dimensions into grids
grid_x, grid_y = np.mgrid[
x.min():x.max():1j*reso_x,
y.min():y.max():1j*reso_y
]
grid_z = si.griddata(
xy, z.values,
(grid_x, grid_y),
method=interp
)
grid_g = si.griddata(
xy, g.values,
(grid_x, grid_y),
method=interp
)
return {
'x' : grid_x,
'y' : grid_y,
'z' : grid_z,
'g' : grid_g,
}
Let's use grids_maker over our list of arrays and get the extrema of each z-level's 4th dimension.
g_mins = []
g_maxs = []
lgrids = {}
for level, arrays_list in planes.items():
lgrids[level] = grids = grids_maker(arrays_list)
g_mins.append(grids['g'].min())
g_maxs.append(grids['g'].max())
Let's create our (all-file unifying) color-scale and show the plot.
# Create the 4th color-rendered dimension
scam = plt.cm.ScalarMappable(
norm=cm.colors.Normalize(min(g_mins), max(g_maxs)),
cmap='jet' # see https://matplotlib.org/examples/color/colormaps_reference.html
)
fig = plt.figure()
ax = fig.gca(projection='3d')
for grids in lgrids.values():
scam.set_array([])
ax.plot_surface(
grids['x'], grids['y'], grids['z'],
facecolors = scam.to_rgba(grids['g']),
antialiased = True,
rstride=1, cstride=1, alpha=None
)
plt.show()
I would be glad to see the result.

Fitting a line through 3D x,y,z scatter plot data

I have a handful of data points that cluster along a line in 3d space. I have the x,y,z data in a csv file that I want to import. I would like to find an equation that represents that line, or the plane perpendicular to that line, or whatever is mathematically correct. These data are independent of each other. Maybe there are better ways to do this than what I tried to do but...
I attempted to replicate an old post here that seemed to be doing exactly what I'm trying to do
Fitting a line in 3D
but it seems that maybe updates over the past decade have left the second part of the code not working? Or maybe I'm just doing something wrong. I've included the entire thing that I frankensteined together from this at the bottom. There are two lines that seem to be giving me a problem.
I've snippeted them out here...
import numpy as np
pts = np.add.accumulate(np.random.random((10,3)))
x,y,z = pts.T
# this will find the slope and x-intercept of a plane
# parallel to the y-axis that best fits the data
A_xz = np.vstack((x, np.ones(len(x)))).T
m_xz, c_xz = np.linalg.lstsq(A_xz, z)[0]
# again for a plane parallel to the x-axis
A_yz = np.vstack((y, np.ones(len(y)))).T
m_yz, c_yz = np.linalg.lstsq(A_yz, z)[0]
# the intersection of those two planes and
# the function for the line would be:
# z = m_yz * y + c_yz
# z = m_xz * x + c_xz
# or:
def lin(z):
x = (z - c_xz)/m_xz
y = (z - c_yz)/m_yz
return x,y
#verifying:
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
fig = plt.figure()
ax = Axes3D(fig)
zz = np.linspace(0,5)
xx,yy = lin(zz)
ax.scatter(x, y, z)
ax.plot(xx,yy,zz)
plt.savefig('test.png')
plt.show()
They return this, but no values...
FutureWarning: rcond parameter will change to the default of machine precision times max(M, N) where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass rcond=None, to keep using the old, explicitly pass rcond=-1.
m_xz, c_xz = np.linalg.lstsq(A_xz, z)[0]
FutureWarning: rcond parameter will change to the default of machine precision times max(M, N) where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass rcond=None, to keep using the old, explicitly pass rcond=-1.
m_yz, c_yz = np.linalg.lstsq(A_yz, z)[0]
I don't know where to go from here. I don't even actually need the plot, I just needed an equation and am ill-equipped to move forward. If anyone knows an easier way to do this, or can point me in the right direction, I'm willing to learn, but I'm very, very lost. Thank you in advance!!
Here is my entire frankensteined code in case that is what is causing the issue.
import pandas as pd
import numpy as np
mydataset = pd.read_csv('line1.csv')
x = mydataset.iloc[:,0]
y = mydataset.iloc[:,1]
z = mydataset.iloc[:,2]
data = np.concatenate((x[:, np.newaxis],
y[:, np.newaxis],
z[:, np.newaxis]),
axis=1)
# Calculate the mean of the points, i.e. the 'center' of the cloud
datamean = data.mean(axis=0)
# Do an SVD on the mean-centered data.
uu, dd, vv = np.linalg.svd(data - datamean)
# Now vv[0] contains the first principal component, i.e. the direction
# vector of the 'best fit' line in the least squares sense.
# Now generate some points along this best fit line, for plotting.
# we want it to have mean 0 (like the points we did
# the svd on). Also, it's a straight line, so we only need 2 points.
linepts = vv[0] * np.mgrid[-100:100:2j][:, np.newaxis]
# shift by the mean to get the line in the right place
linepts += datamean
# Verify that everything looks right.
import matplotlib.pyplot as plt
import mpl_toolkits.mplot3d as m3d
ax = m3d.Axes3D(plt.figure())
ax.scatter3D(*data.T)
ax.plot3D(*linepts.T)
plt.show()
# this will find the slope and x-intercept of a plane
# parallel to the y-axis that best fits the data
A_xz = np.vstack((x, np.ones(len(x)))).T
m_xz, c_xz = np.linalg.lstsq(A_xz, z)[0]
# again for a plane parallel to the x-axis
A_yz = np.vstack((y, np.ones(len(y)))).T
m_yz, c_yz = np.linalg.lstsq(A_yz, z)[0]
# the intersection of those two planes and
# the function for the line would be:
# z = m_yz * y + c_yz
# z = m_xz * x + c_xz
# or:
def lin(z):
x = (z - c_xz)/m_xz
y = (z - c_yz)/m_yz
return x,y
print(x,y)
#verifying:
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
fig = plt.figure()
ax = Axes3D(fig)
zz = np.linspace(0,5)
xx,yy = lin(zz)
ax.scatter(x, y, z)
ax.plot(xx,yy,zz)
plt.savefig('test.png')
plt.show()

As was proposed in the old post you refer to, you could also make use of principal component analysis instead of a least squares approach. For that I suggest sklearn.decomposition.PCA from the sklearn package.
An example can be found below using the csv-file you provided.
import pandas as pd
import numpy as np
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
mydataset = pd.read_csv('line1.csv')
x = mydataset.iloc[:,0]
y = mydataset.iloc[:,1]
z = mydataset.iloc[:,2]
coords = np.array((x, y, z)).T
pca = PCA(n_components=1)
pca.fit(coords)
direction_vector = pca.components_
print(direction_vector)
# Create plot
origin = np.mean(coords, axis=0)
euclidian_distance = np.linalg.norm(coords - origin, axis=1)
extent = np.max(euclidian_distance)
line = np.vstack((origin - direction_vector * extent,
origin + direction_vector * extent))
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(coords[:, 0], coords[:, 1], coords[:,2])
ax.plot(line[:, 0], line[:, 1], line[:, 2], 'r')

You can get rid of the complaint from leastsquares by adding rcond=None like this:
m_xz, c_xz = np.linalg.lstsq(A_xz, z, rcond=None)[0]
Is this the right decision for your situation? I have no idea. But there's more about it in the docs.
When I run your code with your inputs it seems to run just fine and I get values assigned to m_xz, c_xz, etc. If you don't call them explicitly with print('m_xz') (or whatever) then you won't see them.
m_xz
Out[42]: 5.186132604596112
c_xz
Out[43]: 62.5764694106141
Also, you reference your data in kind of two different ways. You get x, y, and z from your csv, but also put it into a numpy array. You can get rid of the duplication and pandas by just using numpy:
data = np.genfromtxt('line1.csv', delimiter=',', skip_header=1)
x = data[:,0]
y = data[:,1]
z = data[:,2]

How do I create a surface plot in python using values from a text file?

I am trying to get a 3D surface plot from a text file that has four coloumns and 700 rows. I have written the code below to generate that plot, but I constantly keep getting the following error:
invalid value encountered in true_divide
z = (x*y*mlim)/(x*y)
raise ValueError("Argument Z must be 2-dimensional.")
ValueError: Argument Z must be 2-dimensional.
Here is my code
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
import numpy as np
f2 = open('openfile.txt', 'r')
i6 = list()
i9 = list()
i10 = list ()
count = 0
while True:
a = f2.readline()
if not a: break
v = a.split()
i6.append(float(v[0]))
i9.append(float(v[1]))
i10.append(float(v[3]))
count = count + 1
mlim = np.array(i6)
ira = np.array(i9)
dec = np.array(i10)
x = ira
y = dec
z = (x*y*mlim)/(x*y)
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_wireframe(x,y,z, rstride=10, cstride=10)
plt.savefig('filesave.png')
plt.close()
How do I solve this?
I have also attached an image of what the actual text file looks like
click here
Thank you.

To plot a 3D surface, a wire frame in your case, you must create a meshgrid (a.k.a a matrix) before so as to map this in you function (x*y*mlim)/(x*y).
The following alteration may fix your problem:
# More of your code above this
x = ira
y = dec
# Create a meshgrid
X, Y = np.meshgrid(x,y)
# Now you can build you z array using the correct mapping
# Note that mlim variable works as an scalar
z = (X*Y*mlim)/(X*Y)
The comment of #Koustav above is pertinent too. It is important check for null/zero values in case of division.

Python irregular x,y data to contour plot on original domain

I have file containing points under the columns "x-cord", "y-cord", "value". These are irregularly spaced. I am trying to make a contour plot of "value" and overlay this over the original domain. I gave up trying to do this in both pgfplots and matlab and thought I would give python a go. An answer in any of these scripts would be fine. The python script is as follows
import numpy as np
from scipy.interpolate import griddata
import matplotlib.pyplot as plt
import numpy.ma as ma
from numpy.random import uniform, seed
from scipy.spatial import ConvexHull
#
# Loading data
filename = "strain.dat"
coordinates = []
x_c = []
y_c = []
z_c = []
xyz = open(filename)
title = xyz.readline()
for line in xyz:
x,y,z = line.split()
coordinates.append([float(x), float(y), float(z)])
x_c.append([float(x)])
y_c.append([float(y)])
z_c.append([float(z)])
xyz.close()
#
# Rehaping and translating data
x_c=np.ravel(np.array(x_c))
y_c=np.ravel(np.array(y_c))
z_c=np.ravel(np.array(z_c))
x_c = x_c-100.0
y_c = y_c-100.0
#
# Checking the convex hull
points=np.column_stack((x_c,y_c))
hull = ConvexHull(points);
plt.plot(points[hull.vertices,0], points[hull.vertices,1], 'r--', lw=2)
plt.scatter(x_c, y_c, marker='o', s=5, zorder=10)
#
# Mapping the irregular data onto a regular grid and plotting
xic = np.linspace(min(x_c), max(x_c), 1000)
yic = np.linspace(min(y_c), max(y_c), 1000)
zic = griddata((x_c, y_c), z_c, (xic[None,:], yic[:,None]))
CS = plt.contour(xic,yic,zic,15,linewidths=0.5,colors='k')
CS = plt.contourf(xic,yic,zic,15,cmap=plt.cm.summer)
plt.colorbar() # draw colorbar
#
#plt.scatter(x_c, y_c, marker='o', s=5, zorder=10)
plt.axis('equal')
plt.savefig('foo.pdf', bbox_inches='tight')
plt.show()
and the output looks like
The problem is that griddata uses a convex hull and this convex hull exceeds the edges of the irregular data. Is there any way to set the values of the griddata points which are outside the edges of the boundary of the original points to zero?
Edit
In the end I threw in the towel and reverted back to Matlab. I'll have to export the data to pgfplots to get a nice plot. The code I came up with was
x = strain.x;
y = strain.y;
z = strain.eps;
% Get the alpha shape (couldn't do this in python easily)
shp = alphaShape(x,y,.001);
% Get the boundary nodes
[bi, xy] = boundaryFacets(shp);
no_grid = 500;
xb=xy(:,1);
yb=xy(:,2);
[X,Y] = ndgrid(linspace(min(x),max(x),no_grid),linspace(min(y),max(y),no_grid));
Z = griddata(x,y,z,X,Y,'v4');
% Got through the regular grid and set the values which are outside the boundary of the original domain to Nans
for j = 1:no_grid
[in,on] = inpolygon(X(:,j),Y(:,j),xb,yb);
Z(~in,j) = NaN;
end
contourf(X,Y,Z,10),axis equal
colorbar
hold on
plot(xb,yb)
axis equal
hold off
Here is the resulting image.
If someone can do something similar in Python I'll happily accept the answer.

I had to plot interpolated data on a complex geometry (see the blue points on figure) P(x,z) (z is the horizontal coordinate). I used mask operations and it worked well. Without mask, the whole square (x=0..1 ; z=0..17.28) is covered by contourf.
## limiting values for geometry
xmax1=0.408
zmin1=6.
xmax2=0.064
zmin2=13.12
xmin=0.
xmax=1.
zmin=0.
zmax=17.28
# Grid for points
x1 = np.arange(xmin,xmax+dx,dx)
z1 = np.arange(zmin,zmax+dz,dz)
zi2,xi2 = np.meshgrid(z1,x1)
mask = (((zi2 > zmin2) & (xi2 > xmax2)) | ((zi2 > zmin1) & (zi2 <= zmin2) & (xi2 > xmax1)))
zim=np.ma.masked_array(zi2,mask)
xim=np.ma.masked_array(xi2,mask)
# Grid for P values
# npz=z coordinates of data, npx is the x coordinates and npp is P values
grid_p = scipy.interpolate.griddata((npz, npx), npp, (zim,xim),method='nearest')
pm=np.ma.masked_array(grid_p,mask)
# plot
plt.contour(zim, xim, pm, 25, linewidths=0.5, colors='k',corner_mask=False)
plt.contourf(zim, xim, pm, 25,vmax=grid_p.max(), vmin=grid_p.min(),corner_mask=False)
plt.colorbar()
# Scatter plot to check
plt.scatter(npz,npr, marker='x', s=2)
plt.show()
enter image description here

Step plot by reading from file

I am a newbie to matplotlib. I am trying to plot step function and having some trouble. Right now I am able to read from the file and plot it as shown below. But the graph in the top is not in steps and the one below is not a proper step. I saw examples to plot step function by giving x & y value. I am not sure how to do it by reading from a file though. Can someone help me?
from pylab import plotfile, show, gca
import matplotlib.pyplot as plt
import matplotlib.cbook as cbook
fname = cbook.get_sample_data('sample.csv', asfileobj=False)
plotfile(fname, cols=(0,1), delimiter=' ')
plotfile(fname, cols=(0,2), newfig=False, delimiter=' ')
plt.show()
Sample inputs(3 columns):
27023927 3 0
27023938 2 0
27023949 3 0
27023961 2 0
27023972 3 0
27023984 2 0
27023995 3 0
27024007 2 0
27024008 2 1
27024018 3 1
27024030 2 1
27024031 2 0
27024041 3 0
27024053 2 0
27024054 2 1
27024098 2 0
Note: I have made the y-axis1 values as 3 & 2 so that this graph can occur in the top and another y-axis2 values 0 & 1 so that it comes in the bottom as shown below
Waveform as it looks now

Essentially your resolution is too low, for the lower plot the steps (except the last one) occur over 1 unit in x, while the steps are about an order of magnitude larger. This gives the appearance of steps while if you zoom in you will see the vertical lines have a non-infinite gradient (true steps change with an infinite gradient).
This is the same problem for both the top and bottom plots. We can easily remedy this by using the step function. You will generally find it easier to import the data, in this example I use the powerful numpy genfromtxt. This loads the data as an array data:
import numpy as np
import matplotlib.pylab as plt
data = np.genfromtxt('test.csv', delimiter=" ")
ax1 = plt.subplot(2,1,1)
ax1.step(data[:,0], data[:,1])
ax2 = plt.subplot(2,1,2)
ax2.step(data[:,0], data[:,2])
plt.show()
If you are new to python then there may be two things to mention, we use two subplots (ax1 and ax2) to plot the data rather than plotting on the same plot (this means you wouldn't need to add values to spatially separate them). We access the elements of the array through the [] this gives the [column, row] with : meaning all columns and and index i being the ith column

I would propose to load the data to a numpy array
import numpy as np
data = np.loadtxt('sample.csv')
And than plot it:
# first point
ax = [data[0,0]]
ay = [data[0,1]]
for i in range(1, data.shape[0]):
if ay[-1] != data[i,1]: # if y value has changed
# add current x and old y
ax.append(data[i,0])
ay.append(ay[-1])
# add current x and current y
ax.append(data[i,0])
ay.append(data[i,1])
import matplotlib.pyplot as plt
plt.plot(ax,ay)
plt.show()
What my solution differs from yours, is that I plot two points for every change in y. The two points produce this 90 degree bend. I Only plot the first curve. Change [?,1] to [?,2] for the second one.

Thanks for the suggestions. I was able to plot it after some research and here is my code,
import csv
import datetime
import matplotlib.pyplot as plt
import numpy as np
import dateutil.relativedelta as rd
import bisect
import scipy as sp
fname = "output.csv"
portfolio_list = []
x = []
a = []
b = []
portfolio = csv.DictReader(open(fname, "r"))
portfolio_list.extend(portfolio)
for data in portfolio_list:
x.append(data['i'])
a.append(data['a'])
b.append(data['b'])
stepList = [0, 1,2,3]
fig = plt.figure(figsize=(20, 10))
ax = fig.add_subplot(111)
plt.step(x, a, 'g', where='post')
plt.step(x, b, 'r', where='post')
plt.show()
and got the image like,