Develop Reference

Matplotlib - How to assign an alpha value to each point on a 3D scatter plot? Radar Data Included

So I am trying to create a 3D scatter plot of radar data, where each point is assigned an alpha value based on the amplitude of the corresponding pixel.
I have done this looping through all x,y,z points and building the scatter plot point by point assigning the alpha values on each iteration. But once the scatter plot is created it is very slow and unable to manipulate the graph without considerable time spent waiting for the plot to update.
Points is a normalised (0 to 1) array.
Here is a link to my data
Data
File preparation:
def normalise0to1(data):
normalised = (data - np.min(data)) / (np.max(data) - np.min(data))
return normalised
Data = np.loadtxt('filepath.txt')
points2D = normalise0to1(Data) #Is (101,400) size
points3D = np.reshape(points2D,(101,20,20)) #Is (101,20,20) size
And the first attempt at creating the 3D scatter plot:
def Scatter_Plot1(points3D):
fig = plt.figure()
ax = fig.add_subplot(projection='3d')
for x in range(0,points3D[1]):
for y in range(0,points3D[2]):
for z in range(0,points3D[0]):
val = points3D[z,x,y]
ax.scatter(x, y, z, alpha=val,c='black',s=1)
plt.show()
This takes a long time to run and is very slow once created.
In 2D, I can do something like this. Bear in mind this is the same array as the 3D 'points' array, but the (20x20) has been flattened to 400. I have provided an image of the flattened array, you can see how it creates an image where intensity is scaled to the alpha value.
def Scatter_Plot2(points2D):
fig = plt.figure()
ax = fig.add_subplot()
x_=np.linspace(0,points2D.shape[1]-1,points2D.shape[1])
y_=np.linspace(0,points2D.shape[0]-1,points2D.shape[0])
x,y = np.meshgrid(x_,y_)
ax.scatter(x,y,alpha=points2D.flatten(),s=1,c='black')
plt.show()
This image is the flattened version of the 3D plot I want to create, where instead of 400 length, it would be (20x20) and overall the 3D shape is (101,20,20).
The problem comes when trying to assign the alpha to a 3D plot, in 2D it seems happy enough when I provide a flattened array to the alpha parameter.
I would like something like this, but whether that's possible is another question..
def Scatter_Plot3(points3D):
fig = plt.figure()
ax = fig.add_subplot(projection='3d')
x_=np.linspace(0,points3D.shape[2]-1,points3D.shape[2])
y_=np.linspace(0,points3D.shape[1]-1,points3D.shape[1])
z_=np.linspace(0,points3D.shape[0]-1,points3D.shape[0])
x,y,z = np.meshgrid(x_,y_,z_)
ax.scatter(x,y,z,alpha=points3D,s=1,c='black')
plt.show()
Results in this image, which seems a bit random in terms of how the alpha is assigned. You would expect to see the dark horizontal lines as in the first picture. What I want may not be possible, I'm open to using another package, perhaps pyqtgraph or mayavi etc. But matplotlib would be preferable.
Thank you!
Edit:
I have achieved something similar to what I would like, though not exactly. I have used the c and cmap inputs. This isn't ideal as I am dealing with a 3D cube and viewing the centre is still difficult but it has correctly mapped a variation to the data. But it doesn't work when I use the alpha parameter.
Notice the 2 main horizontal bands and the dark bit in the centre which is hard to see.
What I need is the same mapping but rather opacity than colour.
c = (points2D.T.flatten())
ax.scatter(x,y,z,s=1,c=c,cmap='viridis_r')

3D scatterplot matplotlib, adding color to plot not working?

I am creating a 3D scatter plot resulting from transforming my data using PCA (3 components). I want my data to be color coded against variable 'phi', which is an array of floats , ranging from 0-360 degrees.
variables x :[44520,3], phi[44520,1)
I get the following error:
ValueError: 'c' argument has 44520 elements, which is not acceptable for use with 'x' with size 44520, 'y' with size 44520.
Any clues?
signalt = dfonf.T # transpose the data as the cell #s are the features and the sampled data is the sample
pca = PCA(n_components=3)
x = pca.fit_transform(signalt)
fig = plt.figure(figsize=(10, 10))
ax = plt.axes(projection="3d")
pcaplot = ax.scatter3D(x[:,0],x[:,1],x[:,2],c=phi,s=0.2)
cbar = fig.colorbar(pcaplot)

The variable c cannot be two dimensional, converted into 1D using
np.squeeze(phi)
This fixed it

Plotting an Obscure Python Graph

I've been trying to work through an interesting problem but have had difficulty finding a proper solution. I am trying to plot columns of heatmaps, with each column potentially varying in row size. The data structure I currently have consists of a nested list, where each list contains various heat values for their points. I'll attach an image below to make this clear.
At this time we have mostly tried to make matplotlib to work, however we haven't been able to produce any of the results we want. Please let me know if you have any idea on what steps we should take next.
Thanks

I think the basic strategy would be to transform your initial array so that you have a rectangular array with the missing values coded as NaN. Then you can simply use imshow to display the heatmap.
I used np.pad to fill the missing values
data = [[0,0,1],[1,0,5,6,0]]
N_cols = len(data)
N_lines = np.max([len(a) for a in data])
data2 = np.array([np.pad(np.array(a, dtype=float), (0,N_lines-len(a)), mode='constant', constant_values=np.nan) for a in data])
fig, ax = plt.subplots()
im = ax.imshow(data2.T, cmap='viridis')
plt.colorbar(im)

How to make a heatmap in python with aggregated/summarized data?

I'm trying to plot some X and Z coordinates on an image to show which parts of the image have higher counts. Y values are height in this case so I am excluding since I want 2D. Since I have many millions of data points, I have grouped by the combinations of X and Z coordinates and counted how many times that value occurred. The data should contain almost all conbinations of X and Z coordinates. It looks something like this (fake data):
I have experimented with matplotlib.pyplot by using the plt.hist2d(x,y) function but it seems like this takes raw data and not already-summarized data like I've got.
Does anyone know if this is possible?
Note: I can figure out the plotting on an image part later, first I'm trying to get the scatter-plot/heatmap to show aggregated data.

I managed to figure this out. After loading in the data in the format of the original post, step one is pivoting the data so you have x values as columns and z values as rows. Then you plot it using seaborn heatmap. See below:
#pivot columns
values = pd.pivot_table(raw, values='COUNT_TICKS', index=['Z_LOC'], columns = ['X_LOC'], aggfunc=np.sum)
plt.figure(figsize=(20, 20))
sns.set(rc={'axes.facecolor':'cornflowerblue', 'figure.facecolor':'cornflowerblue'})
#ax = sns.heatmap(values, vmin=100, vmax=5000, cmap="Oranges", robust = True, xticklabels = x_labels, yticklabels = y_labels, alpha = 1)
ax = sns.heatmap(values,
#vmin=1,
vmax=1000,
cmap="Greens", #BrBG is also good
robust = True,
alpha = 1)
plt.show()

Is there a way to return same length arrays in numpy.hist?

I'm trying to create a histogram plot in python, normalizing with some custom values the y-axis values. For this, I was thinking to do it like this:
import numpy as np
import matplotlib.pyplot as plt
data = np.loadtxt('foo.bar')
fig = plt.figure()
ax = fig.add_subplot(111)
hist=np.histogram(data, bins=(1.0, 1.5 ,2.0,2.5,3.0))
x=[hist[0]*5,hist[1]]
ax.plot(x[0], x[1], 'o')
but of course, the last line gives:
ValueError: x and y must have same first dimension
Is there a way to force np.hist to give the same number of elements for the x[0] and x[1] arrays, for example by deleting the first or last element for one of them?

hist[1] contains the limits in which you have made the histogram. I guess you probably want to get the centers of those intervals, something like:
x = [hist[0], 0.5*(hist[1][1:]+hist[1][:-1])]
and then the plot should be ok, right?

I would imagine it depends on your data source.
Try loading the data as a numpy array, and selecting the range of elements yourself before passing to the histogram function.
e.g.
dataForHistogram = data[0:100][0:100] # Assuming your data is in this kind of structure.