Develop Reference

How do I discretize a continuous function avoiding noise generation (see picture)

I have a continuous input function which I would like to discretize into lets say 5-10 discrete bins between 1 and 0. Right now I am using np.digitize and rescale the output bins to 0-1. Now the problem is that sometime datasets (blue line) yield results like this:
I tried pushing up the number of discretization bins but I ended up keeping the same noise and getting just more increments. As an example where the algorithm worked with the same settings but another dataset:
this is the code I used there NumOfDisc = number of bins
intervals = np.linspace(0,1,NumOfDisc)
discretized_Array = np.digitize(Continuous_Array, intervals)
The red ilne in the graph is not important. The continuous blue line is the on I try to discretize and the green line is the discretized result.The Graphs are created with matplotlyib.pyplot using the following code:
def CheckPlots(discretized_Array, Continuous_Array, Temperature, time, PlotName)
logging.info("Plotting...")
#Setting Axis properties and titles
fig, ax = plt.subplots(1, 1)
ax.set_title(PlotName)
ax.set_ylabel('Temperature [°C]')
ax.set_ylim(40, 110)
ax.set_xlabel('Time [s]')
ax.grid(b=True, which="both")
ax2=ax.twinx()
ax2.set_ylabel('DC Power [%]')
ax2.set_ylim(-1.5,3.5)
#Plotting stuff
ax.plot(time, Temperature, label= "Input Temperature", color = '#c70e04')
ax2.plot(time, Continuous_Array, label= "Continuous Power", color = '#040ec7')
ax2.plot(time, discretized_Array, label= "Discrete Power", color = '#539600')
fig.legend(loc = "upper left", bbox_to_anchor=(0,1), bbox_transform=ax.transAxes)
logging.info("Done!")
logging.info("---")
return
Any Ideas what I could do to get sensible discretizations like in the second case?

The following solution gives the exact result you need.
Basically, the algorithm finds an ideal line, and attempts to replicate it as well as it can with less datapoints. It starts with 2 points at the edges (straight line), then adds one in the center, then checks which side has the greatest error, and adds a point in the center of that, and so on, until it reaches the desired bin count. Simple :)
import warnings
warnings.simplefilter('ignore', np.RankWarning)
def line_error(x0, y0, x1, y1, ideal_line, integral_points=100):
"""Assume a straight line between (x0,y0)->(x1,p1). Then sample the perfect line multiple times and compute the distance."""
straight_line = np.poly1d(np.polyfit([x0, x1], [y0, y1], 1))
xs = np.linspace(x0, x1, num=integral_points)
ys = straight_line(xs)
perfect_ys = ideal_line(xs)
err = np.abs(ys - perfect_ys).sum() / integral_points * (x1 - x0) # Remove (x1 - x0) to only look at avg errors
return err
def discretize_bisect(xs, ys, bin_count):
"""Returns xs and ys of discrete points"""
# For a large number of datapoints, without loss of generality you can treat xs and ys as bin edges
# If it gives bad results, you can edges in many ways, e.g. with np.polyline or np.histogram_bin_edges
ideal_line = np.poly1d(np.polyfit(xs, ys, 50))
new_xs = [xs[0], xs[-1]]
new_ys = [ys[0], ys[-1]]
while len(new_xs) < bin_count:
errors = []
for i in range(len(new_xs)-1):
err = line_error(new_xs[i], new_ys[i], new_xs[i+1], new_ys[i+1], ideal_line)
errors.append(err)
max_segment_id = np.argmax(errors)
new_x = (new_xs[max_segment_id] + new_xs[max_segment_id+1]) / 2
new_y = ideal_line(new_x)
new_xs.insert(max_segment_id+1, new_x)
new_ys.insert(max_segment_id+1, new_y)
return new_xs, new_ys
BIN_COUNT = 25
new_xs, new_ys = discretize_bisect(xs, ys, BIN_COUNT)
plot_graph(xs, ys, new_xs, new_ys, f"Discretized and Continuous comparison, N(cont) = {N_MOCK}, N(disc) = {BIN_COUNT}")
print("Bin count:", len(new_xs))
Moreover, here's my simplified plotting function I tested with.
def plot_graph(cont_time, cont_array, disc_time, disc_array, plot_name):
"""A simplified version of the provided plotting function"""
# Setting Axis properties and titles
fig, ax = plt.subplots(figsize=(20, 4))
ax.set_title(plot_name)
ax.set_xlabel('Time [s]')
ax.set_ylabel('DC Power [%]')
# Plotting stuff
ax.plot(cont_time, cont_array, label="Continuous Power", color='#0000ff')
ax.plot(disc_time, disc_array, label="Discrete Power", color='#00ff00')
fig.legend(loc="upper left", bbox_to_anchor=(0,1), bbox_transform=ax.transAxes)
Lastly, here's the Google Colab

If what I described in the comments is the problem, there are a few options to deal with this:
Do nothing: Depending on the reason you're discretizing, you might want the discrete values to reflect the continuous values accurately
Change the bins: you could shift the bins or change the number of bins, such that relatively 'flat' parts of the blue line stay within one bin, thus giving a flat green line in these parts as well, which would be visually more pleasing like in your second plot.

How to plot scipy.hierarchy.dendrogram using polar coordinates?

I'm trying to adapt the following resources to this question:
Python conversion between coordinates
https://matplotlib.org/gallery/pie_and_polar_charts/polar_scatter.html
I can't seem to get the coordinates to transfer the dendrogram shape over to polar coordinates.
Does anyone know how to do this? I know there is an implementation in networkx but that requires building a graph and then using pygraphviz backend to get the positions.
Is there a way to convert dendrogram cartesian coordinates to polar coordinates with matplotlib and numpy?
import requests
from ast import literal_eval
import matplotlib.pyplot as plt
import numpy as np
def read_url(url):
r = requests.get(url)
return r.text
def cartesian_to_polar(x, y):
rho = np.sqrt(x**2 + y**2)
phi = np.arctan2(y, x)
return(rho, phi)
def plot_dendrogram(icoord,dcoord,figsize, polar=False):
if polar:
icoord, dcoord = cartesian_to_polar(icoord, dcoord)
with plt.style.context("seaborn-white"):
fig = plt.figure(figsize=figsize)
ax = fig.add_subplot(111, polar=polar)
for xs, ys in zip(icoord, dcoord):
ax.plot(xs,ys, color="black")
ax.set_title(f"Polar= {polar}", fontsize=15)
# Load the dendrogram data
string_data = read_url("https://pastebin.com/raw/f953qgdr").replace("\r","").replace("\n","").replace("\u200b\u200b","")
# Convert it to a dictionary (a subset of the output from scipy.hierarchy.dendrogram)
dendrogram_data = literal_eval(string_data)
icoord = np.asarray(dendrogram_data["icoord"], dtype=float)
dcoord = np.asarray(dendrogram_data["dcoord"], dtype=float)
# Plot the cartesian version
plot_dendrogram(icoord,dcoord, figsize=(8,3), polar=False)
# Plot the polar version
plot_dendrogram(icoord,dcoord, figsize=(5,5), polar=True)
I just tried this and it's closer but still not correct:
import matplotlib.transforms as mtransforms
with plt.style.context("seaborn-white"):
fig, ax = plt.subplots(figsize=(5,5))
for xs, ys in zip(icoord, dcoord):
ax.plot(xs,ys, color="black",transform=trans_offset)
ax_polar = plt.subplot(111, projection='polar')
trans_offset = mtransforms.offset_copy(ax_polar.transData, fig=fig)
for xs, ys in zip(icoord, dcoord):
ax_polar.plot(xs,ys, color="black",transform=trans_offset)

You can make the "root" of the tree start in the middle and have the leaves outside. You also have to add more points to the "bar" part for it to look nice and round.
We note that each element of icoord and dcoord (I will call this seg) has four points:
seg[1] seg[2]
+-------------+
| |
+ seg[0] + seg[3]
The vertical bars are fine as straight lines between the two points, but we need more points between seg[1] and seg[2] (the horizontal bar, which will need to become an arc).
This function will add more points in those positions and can be called on both xs and ys in the plotting function:
def smoothsegment(seg, Nsmooth=100):
return np.concatenate([[seg[0]], np.linspace(seg[1], seg[2], Nsmooth), [seg[3]]])
Now we must modify the plotting function to calculate the radial coordinates. Some experimentation has led to the log formula I am using, based on the other answer which also uses log scale. I've left a gap open on the right for the radial labels and done a very rudimentary mapping of the "icoord" coordinates to the radial ones so that the labels correspond to the ones in the rectangular plot. I don't know exactly how to handle the radial dimension. The numbers are correct for the log, but we probably want to map them as well.
def plot_dendrogram(icoord,dcoord,figsize, polar=False):
if polar:
dcoord = -np.log(dcoord+1)
# avoid a wedge over the radial labels
gap = 0.1
imax = icoord.max()
imin = icoord.min()
icoord = ((icoord - imin)/(imax - imin)*(1-gap) + gap/2)*2*numpy.pi
with plt.style.context("seaborn-white"):
fig = plt.figure(figsize=figsize)
ax = fig.add_subplot(111, polar=polar)
for xs, ys in zip(icoord, dcoord):
if polar:
xs = smoothsegment(xs)
ys = smoothsegment(ys)
ax.plot(xs,ys, color="black")
ax.set_title(f"Polar= {polar}", fontsize=15)
if polar:
ax.spines['polar'].set_visible(False)
ax.set_rlabel_position(0)
Nxticks = 10
xticks = np.linspace(gap/2, 1-gap/2, Nxticks)
ax.set_xticks(xticks*np.pi*2)
ax.set_xticklabels(np.round(np.linspace(imin, imax, Nxticks)).astype(int))
Which results in the following figure:

First, I think you might benefit from this question.
Then, let's break down the objective: it is not very clear to me what you want to do, but I assume you want to get something that looks like this
(source, page 14)
To render something like this, you need to be able to render horizontal lines that appear as hemi-circles in polar coordinates. Then, it's a matter of mapping your horizontal lines to polar plot.
First, note that your radius are not normalized in this line:
if polar:
icoord, dcoord = cartesian_to_polar(icoord, dcoord)
you might normalize them by simply remapping icoord to [0;2pi).
Now, let's try plotting something simpler, instead of your complex plot:
icoord, dcoord = np.meshgrid(np.r_[1:10], np.r_[1:4])
# Plot the cartesian version
plot_dendrogram(icoord, dcoord, figsize=(8, 3), polar=False)
# Plot the polar version
plot_dendrogram(icoord, dcoord, figsize=(5, 5), polar=True)
Result is the following:
as you can see, the polar code does not map horizontal lines to semi-circles, therefore that is not going to work. Let's try with plt.polar instead:
plt.polar(icoord.T, dcoord.T)
produces
which is more like what we need. We need to fix the angles first, and then we shall consider that Y coordinate goes inward (while you probably want it going from center to border). It boils down to this
nic = (icoord.T - icoord.min()) / (icoord.max() - icoord.min())
plt.polar(2 * np.pi * nic, -dcoord.T)
which produces the following
Which is similar to what you need. Note that straight lines remain straight, and are not replaced with arcs, so you might want to resample them in your for loop.
Also, you might benefit from single color and log-scale to make reading easier
plt.subplots(figsize=(10, 10))
ico = (icoord.T - icoord.min()) / (icoord.max() - icoord.min())
plt.polar(2 * np.pi * ico, -np.log(dcoord.T), 'b')

Python: Plot residuals on a fitted model

I want to plot the lines (residuals; cyan lines) between data points and the estimated model. Currently I'm doing so by iterating over all data points in my income pandas.DataFrame and adding vertical lines. x, y are the points' coordinates and predicted are the predictions (here the blue line).
plt.scatter(income["Education"], income["Income"], c='red')
plt.ylim(0,100)
for indx, (x, y, _, _, predicted) in income.iterrows():
plt.axvline(x, y/100, predicted/100) # /100 because it needs floats [0,1]
Is there a more efficient way? This doesn't seem like a good approach for more than a few rows.

First of all note that axvline here only works by coincidence. In general the y values taken by axvline are in coordinates relative to the axes, not in data coordinates.
In contrast, vlines uses data coordinates and also has the advantage to accept arrays of values. It will then create a LineCollection, which is more efficient than individual lines.
import matplotlib.pyplot as plt
import numpy as np
x = np.linspace(-1.2,1.2,20)
y = np.sin(x)
dy = (np.random.rand(20)-0.5)*0.5
fig, ax = plt.subplots()
ax.plot(x,y)
ax.scatter(x,y+dy)
ax.vlines(x,y,y+dy)
plt.show()

How to draw "two directions widths line" in matplotlib

How to use matplotlib or pyqtgraph draw plot like this:
Line AB is a two-directions street, green part represents the direction from point A to point B, red part represents B to A, width of each part represents the traffic volume. Widths are measured in point, will not changed at different zoom levels or dpi settings.
This is only an example, in fact I have hunderds of streets. This kind of plot is very common in many traffic softwares. I tried to use matplotlib's patheffect but result is frustrated:
from matplotlib import pyplot as plt
import matplotlib.patheffects as path_effects
x=[0,1,2,3]
y=[1,0,0,-1]
ab_width=20
ba_width=30
fig, axes= plt.subplots(1,1)
center_line, = axes.plot(x,y,color='k',linewidth=2)
center_line.set_path_effects(
[path_effects.SimpleLineShadow(offset=(0, -ab_width/2),shadow_color='g', alpha=1, linewidth=ab_width),
path_effects.SimpleLineShadow(offset=(0, ba_width/2), shadow_color='r', alpha=1, linewidth=ba_width),
path_effects.SimpleLineShadow(offset=(0, -ab_width), shadow_color='k', alpha=1, linewidth=2),
path_effects.SimpleLineShadow(offset=(0, ba_width), shadow_color='k', alpha=1, linewidth=2),
path_effects.Normal()])
axes.set_xlim(-1,4)
axes.set_ylim(-1.5,1.5)
One idea came to me is to take each part of the line as a standalone line, and recalculate it's position when changing zoom level, but it's too complicated and slow.
If there any easy way to use matplotlib or pyqtgraph draw what I want? Any suggestion will be appreciated!

If you can have each independent line, this can be done easily with the fill_between function.
from matplotlib import pyplot as plt
import numpy as np
x=np.array([0,1,2,3])
y=np.array([1,0,0,-1])
y1width=-1
y2width=3
y1 = y + y1width
y2 = y + y2width
fig = plt.figure()
ax = fig.add_subplot(111)
plt.plot(x,y, 'k', x,y1, 'k',x,y2, 'k',linewidth=2)
ax.fill_between(x, y1, y, color='g')
ax.fill_between(x, y2, y, color='r')
plt.xlim(-1,4)
plt.ylim(-3,6)
plt.show()
Here I considered the center line as the reference (thus the negative y1width), but could be done differently. The result is then:
If the lines are 'complicated', eventually intersecting at some point, then the keyword argument interpolate=True must be used to fill the crossover regions properly. Another interesting argument probably useful for your use case is where, to condition the region, for instance, where=y1 < 0. For more information you can check out the documentation.

One way of solving your issue is using filled polygons, some linear algebra and some calculus. The main idea is to draw a polygon along your x and y coordinates and along shifted coordinates to close and fill the polygon.
These are my results:
And here is the code:
from __future__ import division
import numpy
from matplotlib import pyplot, patches
def road(x, y, w, scale=0.005, **kwargs):
# Makes sure input coordinates are arrays.
x, y = numpy.asarray(x, dtype=float), numpy.asarray(y, dtype=float)
# Calculate derivative.
dx = x[2:] - x[:-2]
dy = y[2:] - y[:-2]
dy_dx = numpy.concatenate([
[(y[1] - y[0]) / (x[1] - x[0])],
dy / dx,
[(y[-1] - y[-2]) / (x[-1] - x[-2])]
])
# Offsets the input coordinates according to the local derivative.
offset = -dy_dx + 1j
offset = w * scale * offset / abs(offset)
y_offset = y + w * scale
#
AB = zip(
numpy.concatenate([x + offset.real, x[::-1]]),
numpy.concatenate([y + offset.imag, y[::-1]]),
)
p = patches.Polygon(AB, **kwargs)
# Returns polygon.
return p
if __name__ == '__main__':
# Some plot initializations
pyplot.close('all')
pyplot.ion()
# This is the list of coordinates of each point
x = [0, 1, 2, 3, 4]
y = [1, 0, 0, -1, 0]
# Creates figure and axes.
fig, ax = pyplot.subplots(1,1)
ax.axis('equal')
center_line, = ax.plot(x, y, color='k', linewidth=2)
AB = road(x, y, 20, color='g')
BA = road(x, y, -30, color='r')
ax.add_patch(AB)
ax.add_patch(BA)
The first step in calculating how to offset each data point is by calculating the discrete derivative dy / dx. I like to use complex notation to handle vectors in Python, i.e. A = 1 - 1j. This makes life easier for some mathematical operations.
The next step is to remember that the derivative gives the tangent to the curve and from linear algebra that the normal to the tangent is n=-dy_dx + 1j, using complex notation.
The final step in determining the offset coordinates is to ensure that the normal vector has unity size n_norm = n / abs(n) and multiply by the desired width of the polygon.
Now that we have all the coordinates for the points in the polygon, the rest is quite straightforward. Use patches.Polygon and add them to the plot.
This code allows you also to define if you want the patch on top of your route or below it. Just give a positive or negative value for the width. If you want to change the width of the polygon depending on your zoom level and/or resolution, you adjust the scale parameter. It also gives you freedom to add additional parameters to the patches such as fill patterns, transparency, etc.

Python 3D plotting of measurement data

I have captured 3D measurement data on a sphere (this is an antenna radiation pattern, so the measurement antenna captured the radiation intensity from each phi,theta direction and logged this value as a function of phi,theta).
I am having great difficulty getting the data represented.
I have tried multiple options. This is the last one I am now trying:
import numpy as np
from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
nElevationPoints = 16
nAzimuthPoints = 40
stepSizeRad = 0.05 * np.pi
def r(phi,theta):
radius = 1
return radius
phi = np.arange(0,nAzimuthPoints*stepSizeRad,stepSizeRad)
theta = np.arange(0,nElevationPoints*stepSizeRad,stepSizeRad)
x = (r(phi,theta)*np.outer(r(phi,theta)*np.cos(phi), np.sin(theta)))
y = (-r(phi,theta)*np.outer(np.sin(phi), np.sin(theta)))
z = (r(phi,theta)*np.outer(np.ones(np.size(phi)), np.cos(theta)))
fig = plt.figure(1)
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(x, y, z, rstride=4, cstride=4, color='b')
plt.ioff()
plt.show()
This code in itself is working, and it plots a sphere. Now the thing is, that in accordance with the measurement data, I would actually need the radius not be a constant "1", but corresponding with the radiation intensity measured. So it needs to be a function of phi,theta.
However, as soon as I change the "r" function to anything containing the phi or theta parameter, I get an error about operands that could not be broadcast.
If there's any work around that loops through phi,theta that would be perfectly fine as well.
But I'm stuck now, so I'd appreciate any help :-)
BTW, the reason I went for the above approach is because I couldn't make sense of how the x,y,z should be defined in order to be acceptable to the plot_surface function.
I did manage to generate a scatter plot, by calculating the actual positions (x,y,z) from the phi,theta,intensity data, but this is only a representation by individual points and doesn't generate any well visible antenna radiation pattern plot. For this I assume that a contour plot would be better, but then again I am stuck at either the "r" function call or by understanding how x,y,z should be formatted (the documentation refers to x,y,z needing to be 2D-arrays, but this is beyond my comprehension as x,y,z usually are one dimensional arrays in themselves).
Anyway, looking forward to any help anyone may be willing to give.
-- EDIT --
With #M4rtini 's suggested changes I come to the following:
import numpy as np
from mayavi import mlab
def r(phi,theta):
r = np.sin(phi)**2
return r
phi, theta = np.mgrid[0:2*np.pi:201j, 0:np.pi:101j]
x = r(phi,theta)*np.sin(phi)*np.cos(theta)
y = r(phi,theta)*np.sin(phi)*np.sin(theta)
z = r(phi,theta)*np.cos(phi)
intensity = phi * theta
obj = mlab.mesh(x, y, z, scalars=intensity, colormap='jet')
obj.enable_contours = True
obj.contour.filled_contours = True
obj.contour.number_of_contours = 20
mlab.show()
This works, thanks, #M4rtini, and I now am able to have a phi,theta dependent "r" function.
However, noted that the example now ensures phi and theta to be of the same length (due to the mgrid function). This is not the case in my measurement. When declaring phi and theta separately and of different dimensions, it doesn't work still. So I now will have a look into measurement interpolation.

This might not be the exact answer you were looking for, but if you can accept using intensity values as a mapping of a color, this should work.
Actually, you could probably calculate a specific r here also. But i did not test that.
Using mayavi since it is, in my opinion, far superior than matplotlib for 3D.
import numpy as np
from mayavi import mlab
r = 1.0
phi, theta = np.mgrid[0:np.pi:200j, 0:2*np.pi:101j]
x = r*np.sin(phi)*np.cos(theta)
y = r*np.sin(phi)*np.sin(theta)
z = r*np.cos(phi)
intensity = phi * theta
obj = mlab.mesh(x, y, z, scalars=intensity, colormap='jet')
obj.enable_contours = True
obj.contour.filled_contours = True
obj.contour.number_of_contours = 20
mlab.show()
Output of example script, now this is in a interactive gui. so you can rotate, translate, scale as you please. And even interactively manipulate the data, and the representation options.