Generate probability distribution or smoothing plot from points containing probabilities - python

I have points which include the probability on the y-axis and values on the x-axis, like:
p1 =
[[0.0, 0.0001430560406790707],
[10.0, 6.2797052001508247e-13],
[15.0, 4.8114669550502021e-06],
[20.0, 0.0007443231772534647],
[25.0, 0.00061070912573869406],
[30.0, 0.48116582167944905],
[35.0, 0.24698643991977953],
[40.0, 0.016407283121225951],
[45.0, 0.2557158314329116],
[50.0, 1.1252231121357235e-05],
[55.0, 0.064666668633158647],
[60.0, 1.7631447655837744e-17],
[65.0, 1.1294722466816786e-14],
[70.0, 2.9419020411134367e-16],
[75.0, 3.0887653014525822e-17],
[80.0, 4.4973693062706866e-17],
[85.0, 9.0975358174005147e-15],
[90.0, 1.0758266454985257e-10],
[95.0, 7.2923752473657924e-08],
[100.0, 1.8065366882584036e-08]]
p2 =
[[0.0, 4.1652247577331996e-06],
[10.0, 1.2212829713673957e-06],
[15.0, 6.5906857192417344e-08],
[20.0, 0.00016745946587138236],
[25.0, 0.0054431111796765554],
[30.0, 0.0067575214586160616],
[35.0, 0.00011856110316632124],
[40.0, 0.00032181662132509944],
[45.0, 0.001397981055516994],
[50.0, 0.0027058954834684062],
[55.0, 2.553142406703067e-06],
[60.0, 1.1514033594755017e-08],
[65.0, 0.21961568282994792],
[70.0, 2.4658349829099807e-08],
[75.0, 0.0022850986575076743],
[80.0, 3.5603047823624507e-06],
[85.0, 0.99406392082894734],
[90.0, 0.24399923235645221],
[95.0, 0.0013470125217945798],
[100.0, 0.042582366972883985]]
Now I want to generate a probability distribution from the points, where the x-axis values are (0,10,15,20,...,100) and the y-axis values contain the probabilities (0.00014,....)
When using the plt.plot fuction I get:
plt.plot([item[0] for item in p1],[item[1] for item in p1])
And for p2:
plt.plot([item[0] for item in p2],[item[1] for item in p2])
I want to get a more smooth visualization, like a probability distribution:
And if a probability distribution is not possible, then a smoothing spline:


Scipy's gaussian_kde is often used to smoothly approximate a probability distribution. It sums a gaussian kernel for each input point. Usually individual measurements are used as inputs, but the weights parameter allows working with binned data. The function is normalized to have its integral equal to one.
This approach assumes the values of p1 and p2 are meant as a mean for the segment around each x-value, similar to a histogram. I.e. a step function where the x-values identify the end of each step.
from matplotlib import pyplot as plt
import numpy as np
from scipy.stats import gaussian_kde
p1 = np.array([[0.0, 0.0001430560406790707],
[10.0, 6.2797052001508247e-13],
[15.0, 4.8114669550502021e-06],
[20.0, 0.0007443231772534647],
[25.0, 0.00061070912573869406],
[30.0, 0.48116582167944905],
[35.0, 0.24698643991977953],
[40.0, 0.016407283121225951],
[45.0, 0.2557158314329116],
[50.0, 1.1252231121357235e-05],
[55.0, 0.064666668633158647],
[60.0, 1.7631447655837744e-17],
[65.0, 1.1294722466816786e-14],
[70.0, 2.9419020411134367e-16],
[75.0, 3.0887653014525822e-17],
[80.0, 4.4973693062706866e-17],
[85.0, 9.0975358174005147e-15],
[90.0, 1.0758266454985257e-10],
[95.0, 7.2923752473657924e-08],
[100.0, 1.8065366882584036e-08]])
p2 = np.array([[0.0, 4.1652247577331996e-06],
[10.0, 1.2212829713673957e-06],
[15.0, 6.5906857192417344e-08],
[20.0, 0.00016745946587138236],
[25.0, 0.0054431111796765554],
[30.0, 0.0067575214586160616],
[35.0, 0.00011856110316632124],
[40.0, 0.00032181662132509944],
[45.0, 0.001397981055516994],
[50.0, 0.0027058954834684062],
[55.0, 2.553142406703067e-06],
[60.0, 1.1514033594755017e-08],
[65.0, 0.21961568282994792],
[70.0, 2.4658349829099807e-08],
[75.0, 0.0022850986575076743],
[80.0, 3.5603047823624507e-06],
[85.0, 0.99406392082894734],
[90.0, 0.24399923235645221],
[95.0, 0.0013470125217945798],
[100.0, 0.042582366972883985]])
x = np.linspace(0, 100, 1000)
fig, axes = plt.subplots(ncols=2)
for ax, p in zip(axes, [p1, p2]):
p[0, 0] = 5.0 # let each x-value be the end of a segment
ax.step(p[:,0], p[:,1], color='dodgerblue', lw=1, ls=':', where='pre')
ax2 = ax.twinx()
kde = gaussian_kde(p[:,0]-2.5, bw_method=.25, weights=p[:,1])
ax2.plot(x, kde(x), color='crimson')
plt.show()

Related

How to sort boundary boxes from right to left and top to bottom

I have a use case where I want to sort boundary boxes from the top right to the bottom left.
args:
dt_boxes(array): detected text boxes with shape [4, 2]
return:
sorted boxes(array) with shape [4, 2]
Example 1:
[[[258.0, 52.0], [329.0, 46.0], [329.0, 72.0], [260.0, 76.0]], [[91.0, 32.0], [174.0, 43.0], [175.0, 68.0], [90.0, 64.0]], [[182.0, 45.0], [266.0, 42.0], [266.0, 69.0], [184.0, 74.0]], [[16.0, 41.0], [97.0, 39.0], [98.0, 64.0], [17.0, 69.0]], [[226.0, 4.0], [296.0, 4.0], [296.0, 32.0], [227.0, 35.0]], [[288.0, 9.0], [331.0, 2.0], [331.0, 36.0], [289.0, 39.0]]]
Example 2:
[[[224.0, 85.0], [381.0, 83.0], [381.0, 128.0], [223.0, 126.0]], [[412.0, 88.0], [544.0, 81.0], [545.0, 129.0], [413.0, 134.0]], [[291.0, 18.0], [357.0, 18.0], [357.0, 56.0], [292.0, 68.0]], [[122.0, 12.0], [295.0, 11.0], [296.0, 57.0], [125.0, 64.0]], [[350.0, 22.0], [435.0, 11.0], [435.0, 55.0], [351.0, 66.0]], [[442.0, 15.0], [538.0, 11.0], [539.0, 49.0], [442.0, 57.0]], [[9.0, 12.0], [125.0, 8.0], [127.0, 54.0], [10.0, 63.0]]]
I have used the sort function like this to sort using the 2nd element (Top Right) and 3rd Element (Bottom Right) in the array but it doesn't provide the correct sorting
Results to be achieved :
Boundaries are upper left, upper right, lower right, and lower left.
sorted_boxes = sorted(dt_boxes, key=lambda x: (x[1][1], x[1][0]),reverse=True)

Unexpected dbscan result

i have implemented a code to cluster simple array of point's but the results are unexpected.
code:
from pandas import DataFrame
import matplotlib.pyplot as plt
import matplotlib
from sklearn.cluster import DBSCAN
x1 = [[10.0, 1.0], [10.0, 2.0], [10.0, 10.0], [10.0, 10.0], [10.0, 23.0], [10.0, 22.0]]
x2 = [[20.0, 2.0], [20.0, 15.0], [20.0, 26.0], [20.0, 13.0], [20.0, 32.0], [20.0, 35.0]]
x3 = [[30.0, 25.0], [30.0, 28.0], [30.0, 17.0], [30.0, 16.0], [30.0, 15.0], [30.0, 38.0]]
x4 = [[40.0, 1.0], [40.0, 2.0], [40.0, 16.0], [40.0, 41.0], [40.0, 40.0], [40.0, 39.0]]
x5 = [[60.0, 1.0], [60.0, 10.0], [60.0, 12.0], [60.0, 32.0], [60.0, 33.0], [60.0, 50.0]]
df1 = DataFrame(data= x1)
df2 = DataFrame(data= x2)
df3 = DataFrame(data= x3)
df4 = DataFrame(data= x4)
df5 = DataFrame(data= x5)
data = df5
dbscan_opt=DBSCAN(eps=1,min_samples=2)
dbscan_opt.fit(data[[0,1]])
data['DBSCAN_opt_labels']=dbscan_opt.labels_
data['DBSCAN_opt_labels'].value_counts()
# Plotting the resulting clusters
colors=['purple','red','blue','green']
plt.figure(figsize=(6,6))
plt.scatter(data[0],data[1],c=data['DBSCAN_opt_labels'],cmap=matplotlib.colors.ListedColormap(colors),s=60)
plt.title('DBSCAN Clustering',fontsize=20)
plt.xlabel('Feature 1',fontsize=14)
plt.ylabel('Feature 2',fontsize=14)
plt.show()
results:
Red circle means that i expect a cluster but the algorithm detect them as noise points..
How to fix parameter's of DBSCAN to detect them as well?

i found the problem. limited color list! i should add more colors because DBSCAN generates new cluster but with same color and that was my mistake.
so this is working fine now:
dbscan_opt=DBSCAN(eps=3.0,min_samples=1).fit(data[[0,1]])
data['DBSCAN_opt_labels']=dbscan_opt.labels_
data['DBSCAN_opt_labels'].value_counts()
print(data)
# Plotting the resulting clusters
colors=['purple','red','blue','green', 'magenta', 'teal', 'black']
plt.figure(figsize=(6,6))
plt.scatter(data[0],data[1],c=data['DBSCAN_opt_labels'],cmap=matplotlib.colors.ListedColormap(colors),s=60)
plt.title('DBSCAN Clustering',fontsize=20)
plt.xlabel('Feature 1',fontsize=14)
plt.ylabel('Feature 2',fontsize=14)
plt.show()

Create video from python plots to look like a moving probability distribution

I generate i plots with the code:
for i in range(len(hlist)):
p1 = np.array(hlist[i])
kde = gaussian_kde(p1[:,0], bw_method=.4, weights=p1[:,1])
x = np.linspace(0, 100, 1000)
plt.plot(x, kde(x), color='crimson')
plt.ylim([minmin, maxmax])
plt.show()
The x-axis and the y-axis of all plots are the same
So I want to make a video from all plots and if its possible with transition between each plots so it looks like a moving probability distribution
Full hlist:
hlist = np.array([[[0.0, 0.9003079895267837],
[10.0, 0.000654274453636099],
[15.0, 4.8062543883990662e-05],
[20.0, 0.0078540048399245721],
[25.0, 0.00030775066073687057],
[30.0, 0.0069461270425554486],
[35.0, 0.00038747043199781708],
[40.0, 0.0040570022317705538],
[45.0, 0.01242208947573863],
[50.0, 0.00042730445196067818],
[55.0, 9.2094566445921082e-05],
[60.0, 2.3533619494819096e-17],
[65.0, 6.8535505965721361e-15],
[70.0, 2.3163309687227612e-12],
[75.0, 4.7244481305730235e-14],
[80.0, 3.946900273532476e-15],
[85.0, 1.387482842660183e-12],
[90.0, 1.169953999346581e-10],
[95.0, 9.405541221977168e-06],
[100.0, 0.00020815494180476587]],
[[0.0, 0.093092478749998001],
[10.0, 1.4053195886715078e-07],
[15.0, 2.4433977582117203e-05],
[20.0, 0.00082219857285037044],
[25.0, 2.6948063300386935e-05],
[30.0, 0.093959576222220995],
[35.0, 0.0001656191073324495],
[40.0, 0.0029118221893314833],
[45.0, 0.01564108273403109],
[50.0, 1.6648778174359671e-05],
[55.0, 2.8100410030581244e-07],
[60.0, 6.0753841480616879e-17],
[65.0, 9.8080692203972863e-15],
[70.0, 6.9694730549144334e-17],
[75.0, 5.8326312613461052e-15],
[80.0, 2.4139075497471945e-16],
[85.0, 5.9084558298527006e-13],
[90.0, 1.6711579730178305e-10],
[95.0, 1.5864098433225844e-08],
[100.0, 2.2572886039178069e-08]],
[[0.0, 4.7418107222805056e-05],
[10.0, 0.72289205649751387],
[15.0, 3.308651897500352e-07],
[20.0, 0.076673642863552008],
[25.0, 0.0001501767582823212],
[30.0, 0.03027984877348755],
[35.0, 0.00011292381988867554],
[40.0, 0.00016702076039256828],
[45.0, 0.00081579620071354846],
[50.0, 1.0458962769769357e-05],
[55.0, 5.5802811319507686e-08],
[60.0, 4.1253613398572781e-19],
[65.0, 9.4051858824518925e-18],
[70.0, 1.3368786287741004e-14],
[75.0, 5.9931111040137393e-14],
[80.0, 3.2539929633588511e-15],
[85.0, 1.047255336132784e-15],
[90.0, 1.4017813965595768e-12],
[95.0, 2.8511569193363785e-09],
[100.0, 1.0183773063292469e-10]],
[[0.0, 0.0045609821258978814],
[10.0, 0.0001917787950273313],
[15.0, 0.99771505193318455],
[20.0, 0.0055938427174407005],
[25.0, 0.013201240944903663],
[30.0, 0.018570016510629072],
[35.0, 0.012700626279918454],
[40.0, 0.019933344753975846],
[45.0, 0.15619609747558014],
[50.0, 0.031265415297255877],
[55.0, 0.42528804082092103],
[60.0, 1.3054822476679266e-18],
[65.0, 8.152405640596557e-18],
[70.0, 1.4328231368499457e-17],
[75.0, 1.3158371543509022e-15],
[80.0, 2.0369465623715189e-18],
[85.0, 1.1462339774892004e-13],
[90.0, 4.7664892870840778e-13],
[95.0, 7.2181814960747638e-06],
[100.0, 0.0011762750538733889]],
[[0.0, 1.4364539307817039e-05],
[10.0, 0.0002382809395285733],
[15.0, 1.1335942314786911e-05],
[20.0, 0.52821149082545871],
[25.0, 0.1355189607672023],
[30.0, 0.0013765391762318477],
[35.0, 0.01044957461521267],
[40.0, 0.00056647346484793787],
[45.0, 0.00076442899303849577],
[50.0, 0.032497831121074787],
[55.0, 0.00032304336212375635],
[60.0, 8.7317718742465198e-16],
[65.0, 6.539726198592529e-12],
[70.0, 9.2846001586370999e-08],
[75.0, 1.1275715667447136e-16],
[80.0, 9.3449145004368205e-16],
[85.0, 7.6700176188571133e-13],
[90.0, 3.6113266426134806e-09],
[95.0, 8.8657633159259348e-07],
[100.0, 1.0088131613242799e-08]],
[[0.0, 0.00010259026745556786],
[10.0, 2.8416604131363134e-05],
[15.0, 1.5265724585429075e-08],
[20.0, 0.00035572734341432835],
[25.0, 0.74774814405543055],
[30.0, 0.20170120434943195],
[35.0, 0.31688948032690523],
[40.0, 0.016495873067229414],
[45.0, 0.019452385467953772],
[50.0, 0.0063096344918544449],
[55.0, 1.4586177114635856e-07],
[60.0, 1.2622740425026807e-15],
[65.0, 1.6230117141451513e-13],
[70.0, 2.7328268483555935e-15],
[75.0, 5.9099377394502244e-17],
[80.0, 1.0947896137035609e-17],
[85.0, 2.2319783866749845e-12],
[90.0, 7.3069777072490827e-10],
[95.0, 1.2982219252486507e-07],
[100.0, 9.2703590554295812e-09]],
[[0.0, 5.6467274191670147e-06],
[10.0, 1.9311914301356571e-15],
[15.0, 0.00030373592016130291],
[20.0, 0.37222022990454917],
[25.0, 7.1237862206873578e-07],
[30.0, 0.016227799829528728],
[35.0, 0.33096651567634761],
[40.0, 0.00014104696271763757],
[45.0, 0.001730151916132917],
[50.0, 0.00019647094586097279],
[55.0, 1.61437439554684e-14],
[60.0, 7.1139359284635841e-65],
[65.0, 8.3871906097721264e-63],
[70.0, 1.2336993844192023e-53],
[75.0, 5.0051598531858931e-64],
[80.0, 2.9758606051614185e-62],
[85.0, 2.8460978574131379e-51],
[90.0, 2.8677742500204446e-37],
[95.0, 3.7194535849027315e-27],
[100.0, 2.3439805162091607e-24]],
[[0.0, 0.0001430560406790707],
[10.0, 6.2797052001508247e-13],
[15.0, 4.8114669550502021e-06],
[20.0, 0.0007443231772534647],
[25.0, 0.00061070912573869406],
[30.0, 0.48116582167944905],
[35.0, 0.24698643991977953],
[40.0, 0.016407283121225951],
[45.0, 0.2557158314329116],
[50.0, 1.1252231121357235e-05],
[55.0, 0.064666668633158647],
[60.0, 1.7631447655837744e-17],
[65.0, 1.1294722466816786e-14],
[70.0, 2.9419020411134367e-16],
[75.0, 3.0887653014525822e-17],
[80.0, 4.4973693062706866e-17],
[85.0, 9.0975358174005147e-15],
[90.0, 1.0758266454985257e-10],
[95.0, 7.2923752473657924e-08],
[100.0, 1.8065366882584036e-08]],
[[0.0, 8.4950625072720679e-07],
[10.0, 1.1421138441091606e-09],
[15.0, 4.5866299179754217e-05],
[20.0, 1.5172948619703533e-06],
[25.0, 0.00052404930089682738],
[30.0, 0.0013085043037665261],
[35.0, 0.0011664239519834339],
[40.0, 0.82750281205997822],
[45.0, 0.10132991062898908],
[50.0, 0.0085433968529198406],
[55.0, 1.4344253354768514e-09],
[60.0, 4.1465759737420333e-63],
[65.0, 8.8542318068147446e-61],
[70.0, 6.5220929006588264e-57],
[75.0, 4.8147063172017478e-62],
[80.0, 1.6649888561929801e-54],
[85.0, 5.8101309745184615e-51],
[90.0, 1.0489469511522901e-34],
[95.0, 1.4172891722901137e-26],
[100.0, 7.7737342215176175e-21]],
[[0.0, 3.1632028979839915e-05],
[10.0, 0.22111326362785799],
[15.0, 5.0749184816502276e-05],
[20.0, 0.0037647084552139601],
[25.0, 0.06595171605012351],
[30.0, 0.020353996100110065],
[35.0, 0.064862755938758426],
[40.0, 0.063464515818315706],
[45.0, 0.28662321979050187],
[50.0, 0.29871027240136533],
[55.0, 0.0080037249999781925],
[60.0, 4.7989170718538567e-08],
[65.0, 7.1337119197448125e-13],
[70.0, 2.1156334756020827e-14],
[75.0, 5.2206045634456785e-13],
[80.0, 3.4493997391226592e-14],
[85.0, 1.0500710333351341e-12],
[90.0, 3.6230960591527929e-10],
[95.0, 5.4508034466239748e-07],
[100.0, 1.113734841284331e-08]],
[[0.0, 2.1072826422141414e-05],
[10.0, 5.2005089026009156e-06],
[15.0, 0.00035632479168291582],
[20.0, 0.00020117324964893726],
[25.0, 0.025321662490265816],
[30.0, 0.019375041125499575],
[35.0, 0.0072489358231882196],
[40.0, 0.027646703948782582],
[45.0, 0.061781297188577035],
[50.0, 0.61650443678873623],
[55.0, 6.2548100028575296e-10],
[60.0, 4.3624321149810504e-14],
[65.0, 2.1028732991018519e-15],
[70.0, 7.8258766740408378e-09],
[75.0, 2.3541542464960154e-15],
[80.0, 7.1311782140494435e-18],
[85.0, 7.9730756880702952e-14],
[90.0, 8.2168425453267622e-12],
[95.0, 2.7060634831745863e-07],
[100.0, 2.5445845858147043e-06]],
[[0.0, 5.2908114997246604e-05],
[10.0, 0.01017193185840104],
[15.0, 1.9357672782766133e-07],
[20.0, 0.00033902736145621922],
[25.0, 2.6389276490704271e-05],
[30.0, 0.018976734061801625],
[35.0, 0.00019847993445041843],
[40.0, 0.0003987651813445142],
[45.0, 0.00077280097799644102],
[50.0, 7.0623545425863526e-06],
[55.0, 0.49987468099653093],
[60.0, 2.2781104904754321e-10],
[65.0, 3.6325273589160318e-08],
[70.0, 1.3537824953350784e-07],
[75.0, 1.6173509221250697e-06],
[80.0, 1.4416343176395238e-08],
[85.0, 4.1563319348145668e-08],
[90.0, 3.889471307678879e-07],
[95.0, 5.5104492670400779e-05],
[100.0, 2.7151502910109198e-05]],
[[0.0, 6.7973224793022393e-05],
[10.0, 2.4433844497448081e-05],
[15.0, 3.1104921306578064e-06],
[20.0, 0.00034408444849371377],
[25.0, 0.00023308492218191118],
[30.0, 0.011131477320772989],
[35.0, 0.00065907967432373565],
[40.0, 0.0021975971207933783],
[45.0, 0.0064663180932303744],
[50.0, 0.00022182805626473468],
[55.0, 3.0364808672527216e-05],
[60.0, 0.5088867111766856],
[65.0, 0.022878262885332335],
[70.0, 4.5920971324139236e-06],
[75.0, 0.0022218172448998821],
[80.0, 6.2738973247353222e-07],
[85.0, 0.00029845009911656876],
[90.0, 0.00010489918264551884],
[95.0, 0.00060753276898116491],
[100.0, 0.00022654293893088778]],
[[0.0, 3.9706684562645418e-06],
[10.0, 5.1270388737533639e-06],
[15.0, 5.262507305695827e-08],
[20.0, 2.5999077691490454e-05],
[25.0, 2.7813624200874083e-05],
[30.0, 0.0031991625978026266],
[35.0, 7.9249804143628584e-05],
[40.0, 0.00041177406702603777],
[45.0, 0.0011392701023576817],
[50.0, 8.0323465327795955e-06],
[55.0, 4.8154250431354482e-08],
[60.0, 0.49105543193823659],
[65.0, 0.7287168056069141],
[70.0, 0.001517224119720642],
[75.0, 0.011333832180200976],
[80.0, 1.72411085405819e-06],
[85.0, 0.0024773347817556406],
[90.0, 0.0051538392104878905],
[95.0, 0.0012157783163112316],
[100.0, 6.9627245730982824e-06]],
[[0.0, 0.0002912083712195997],
[10.0, 0.00010000827323660753],
[15.0, 1.3045847050921363e-05],
[20.0, 0.00076214254175277312],
[25.0, 0.00069053586992726265],
[30.0, 0.019496501720614564],
[35.0, 0.0015575117915095839],
[40.0, 0.0061672295672820353],
[45.0, 0.024249792592964857],
[50.0, 0.00032644214446402552],
[55.0, 7.7266514916862301e-06],
[60.0, 7.7205272501591181e-06],
[65.0, 0.017897081479520592],
[70.0, 0.99798778384755971],
[75.0, 1.8624562046616415e-05],
[80.0, 0.0011212071784075478],
[85.0, 1.4365766090813676e-05],
[90.0, 0.0032064950461695454],
[95.0, 0.0012746996733909491],
[100.0, 0.002184674454638652]],
[[0.0, 1.4875831960944082e-05],
[10.0, 0.044356141901610378],
[15.0, 6.2599114973306784e-07],
[20.0, 0.0001137512258409283],
[25.0, 0.00074658171939148493],
[30.0, 0.0026392679644018989],
[35.0, 0.00016209963633384973],
[40.0, 0.00078042928094971609],
[45.0, 0.0036289420011120584],
[50.0, 0.0015766136061278139],
[55.0, 1.0905709141111546e-05],
[60.0, 2.7323842104187985e-06],
[65.0, 3.3133452546150147e-05],
[70.0, 9.4042250154063172e-07],
[75.0, 0.97971055505602644],
[80.0, 0.072870311800762033],
[85.0, 0.00053021457261792214],
[90.0, 0.022423571257812192],
[95.0, 0.00070111679082283159],
[100.0, 3.6497263003914267e-05]],
[[0.0, 3.5864031562522839e-05],
[10.0, 8.3611261574776709e-07],
[15.0, 3.1900347488304889e-06],
[20.0, 0.00018123337202990332],
[25.0, 0.0002136705754060659],
[30.0, 0.0057251370200655845],
[35.0, 0.0012906924788080578],
[40.0, 0.0025502956186607153],
[45.0, 0.01424309136060831],
[50.0, 6.8216937833192329e-05],
[55.0, 5.6413951413567533e-06],
[60.0, 3.395689693128361e-05],
[65.0, 0.0032457520420234795],
[70.0, 8.3915675399849457e-09],
[75.0, 0.00013330080558625364],
[80.0, 0.91501828269209085],
[85.0, 0.0020937770133090659],
[90.0, 0.0012871409503378462],
[95.0, 0.00014758805743257718],
[100.0, 0.00086787665688324784]],
[[0.0, 4.1652247577331996e-06],
[10.0, 1.2212829713673957e-06],
[15.0, 6.5906857192417344e-08],
[20.0, 0.00016745946587138236],
[25.0, 0.0054431111796765554],
[30.0, 0.0067575214586160616],
[35.0, 0.00011856110316632124],
[40.0, 0.00032181662132509944],
[45.0, 0.001397981055516994],
[50.0, 0.0027058954834684062],
[55.0, 2.553142406703067e-06],
[60.0, 1.1514033594755017e-08],
[65.0, 0.21961568282994792],
[70.0, 2.4658349829099807e-08],
[75.0, 0.0022850986575076743],
[80.0, 3.5603047823624507e-06],
[85.0, 0.99406392082894734],
[90.0, 0.24399923235645221],
[95.0, 0.0013470125217945798],
[100.0, 0.042582366972883985]],
[[0.0, 5.5827483149344828e-06],
[10.0, 4.2094534384536198e-07],
[15.0, 5.723201102401161e-08],
[20.0, 4.1809740042541515e-05],
[25.0, 0.00049057556903029373],
[30.0, 0.0073861106939193556],
[35.0, 0.00025382321327730041],
[40.0, 0.0013088484764716428],
[45.0, 0.0026974167667276621],
[50.0, 3.069464890989831e-05],
[55.0, 5.8148299885777761e-06],
[60.0, 7.4621165800794304e-07],
[65.0, 0.002374761247356731],
[70.0, 7.989880181871685e-05],
[75.0, 0.0042580794657908335],
[80.0, 0.010965726532792501],
[85.0, 0.00032799281228739028],
[90.0, 0.72335894920682331],
[95.0, 0.0028560935416300647],
[100.0, 0.00023222597525614469]],
[[0.0, 1.1150563179140846e-05],
[10.0, 3.9137754880943888e-06],
[15.0, 2.6540828280675093e-07],
[20.0, 0.00030085951063170473],
[25.0, 0.00099029668428149789],
[30.0, 0.0040087490203686207],
[35.0, 0.00086288885669378408],
[40.0, 0.0012383336640490626],
[45.0, 0.0024569535179785368],
[50.0, 5.3078640889812177e-05],
[55.0, 3.655855476404959e-05],
[60.0, 1.135303263695363e-05],
[65.0, 0.0052337161196729541],
[70.0, 0.00038799120104653003],
[75.0, 6.6611573170762462e-07],
[80.0, 1.551926769812043e-05],
[85.0, 3.0383118936603649e-05],
[90.0, 0.00045596510625991982],
[95.0, 0.97142691361436007],
[100.0, 0.10797398637691005]],
[[0.0, 0.0011842207743420531],
[10.0, 0.00021255186566113124],
[15.0, 0.0014186746952926585],
[20.0, 0.0012807740120210007],
[25.0, 0.0017758699839103857],
[30.0, 0.02941486302872582],
[35.0, 0.0028808476159808511],
[40.0, 0.0053310120235299015],
[45.0, 0.030475142227338822],
[50.0, 0.00050901345787313437],
[55.0, 0.0016516486463798513],
[60.0, 1.288101329664901e-06],
[65.0, 4.7680039665889616e-06],
[70.0, 2.1300407821392148e-05],
[75.0, 3.6408560648746925e-05],
[80.0, 3.0263064939939249e-06],
[85.0, 0.00016351943738745334],
[90.0, 9.5136297575506478e-06],
[95.0, 0.020349612775663113],
[100.0, 0.84447466931781623]]])

With plt.savefig you can create an individual .png file from a plot. To get a smooth transition, you can interpolate the plots in a number of steps. To convert the .png-files to an animation, ffmpeg is a very useful tool.
Here is some code starting from the data of your previous question:
from matplotlib import pyplot as plt
import numpy as np
from scipy.stats import gaussian_kde
p1 = np.array([[5.0, 0.0001430560406790707], [10.0, 6.2797052001508247e-13], [15.0, 4.8114669550502021e-06],
[20.0, 0.0007443231772534647], [25.0, 0.00061070912573869406], [30.0, 0.48116582167944905],
[35.0, 0.24698643991977953], [40.0, 0.016407283121225951], [45.0, 0.2557158314329116],
[50.0, 1.1252231121357235e-05], [55.0, 0.064666668633158647], [60.0, 1.7631447655837744e-17],
[65.0, 1.1294722466816786e-14], [70.0, 2.9419020411134367e-16], [75.0, 3.0887653014525822e-17],
[80.0, 4.4973693062706866e-17], [85.0, 9.0975358174005147e-15], [90.0, 1.0758266454985257e-10],
[95.0, 7.2923752473657924e-08], [100.0, 1.8065366882584036e-08]])
p2 = np.array([[5.0, 4.1652247577331996e-06], [10.0, 1.2212829713673957e-06], [15.0, 6.5906857192417344e-08],
[20.0, 0.00016745946587138236], [25.0, 0.0054431111796765554], [30.0, 0.0067575214586160616],
[35.0, 0.00011856110316632124], [40.0, 0.00032181662132509944], [45.0, 0.001397981055516994],
[50.0, 0.0027058954834684062], [55.0, 2.553142406703067e-06], [60.0, 1.1514033594755017e-08],
[65.0, 0.21961568282994792], [70.0, 2.4658349829099807e-08], [75.0, 0.0022850986575076743],
[80.0, 3.5603047823624507e-06], [85.0, 0.99406392082894734], [90.0, 0.24399923235645221],
[95.0, 0.0013470125217945798], [100.0, 0.042582366972883985]])
x = np.linspace(0, 100, 1000)
hlist = [p1, p2, p1]
kdelist = [gaussian_kde(p[:, 0], bw_method=.4, weights=p[:, 1]) for p in hlist]
steps = 20
fig, ax = plt.subplots()
for i in range(len(kdelist) - 1):
for s in range(steps + 1):
plt.plot(x, kdelist[i](x) * s / steps + kdelist[i + 1](x) * (1 - s / steps), color='crimson')
plt.ylim(0, 0.065)
plt.savefig(f'kde_{i*(steps+1)+s:04d}.png')
plt.cla() # needed to remove the plot because savefig doesn't clear it
An example way to convert the .png files to a .mp4 animation and then to an animated gif:
ffmpeg -i kde_%04d.png -c:v libx264 -r 25 -pix_fmt yuv420p out.mp4
ffmpeg -i out.mp4 out.gif

Constructing a 2d interpolator given scattered input data

I have three lists as follows:
x = [100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0]
y = [300.0, 300.0, 300.0, 300.0, 500.0, 500.0, 500.0, 500.0, 700.0, 700.0, 700.0, 700.0, 1000.0, 1000.0, 1000.0, 1000.0, 1500.0, 1500.0, 1500.0, 1500.0, 2000.0, 2000.0, 2000.0, 2000.0, 3000.0, 3000.0, 3000.0, 3000.0, 5000.0, 5000.0, 5000.0, 5000.0, 7500.0, 7500.0, 7500.0, 75000.0, 10000.0, 10000.0, 10000.0, 10000.0]
z = [100.0, 95.0, 87.5, 77.5, 60.0, 57.0, 52.5, 46.5, 40.0, 38.0, 35.0, 31.0, 30.0, 28.5, 26.25, 23.25, 23.0, 21.85, 20.125, 17.825, 17.0, 16.15, 14.875, 13.175, 13.0, 12.35, 11.375, 10.075, 10.0, 9.5, 8.75, 7.75, 7.0, 6.65, 6.125, 5.425, 5.0, 4.75, 4.375, 3.875]
Each entry of each list is read as a point so point 0 is (100,300,100) point 1 is (75,300,95) and so on.
I am trying to do 2d interpolation, so that I can compute a z value for any given input (x0, y0) point.
I was reading that using meshgrid I can interpolate with RegularGridInterpolator from scipy but I am not sure how to set it up when I do:
x_,y_,z_ = np.meshgrid(x,y,z) # both indexing ij or xy
I don't get values for x_,y_,z_ that make sense and I am not sure how to go from there.
I am trying to use the data points I have above to find intermediate values so something similar to scipy's interp1d where
f = interp1d(x, y, kind='cubic')
where I can later call f(any (x,y) point within range) and get the corresponding z value.

You need 2d interpolation over scattered data. I'd default to using scipy.interpolate.griddata in this case, but you seem to want a callable interpolator, whereas griddata needs a given set of points onto which it will interpolate.
Not to worry: griddata with 2d cubic interpolation uses a CloughTocher2DInterpolator. So we can do exactly that:
import numpy as np
import scipy.interpolate as interp
x = [100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0, 100.0, 75.0, 50.0, 0.0]
y = [300.0, 300.0, 300.0, 300.0, 500.0, 500.0, 500.0, 500.0, 700.0, 700.0, 700.0, 700.0, 1000.0, 1000.0, 1000.0, 1000.0, 1500.0, 1500.0, 1500.0, 1500.0, 2000.0, 2000.0, 2000.0, 2000.0, 3000.0, 3000.0, 3000.0, 3000.0, 5000.0, 5000.0, 5000.0, 5000.0, 7500.0, 7500.0, 7500.0, 75000.0, 10000.0, 10000.0, 10000.0, 10000.0]
z = [100.0, 95.0, 87.5, 77.5, 60.0, 57.0, 52.5, 46.5, 40.0, 38.0, 35.0, 31.0, 30.0, 28.5, 26.25, 23.25, 23.0, 21.85, 20.125, 17.825, 17.0, 16.15, 14.875, 13.175, 13.0, 12.35, 11.375, 10.075, 10.0, 9.5, 8.75, 7.75, 7.0, 6.65, 6.125, 5.425, 5.0, 4.75, 4.375, 3.875]
interpolator = interp.CloughTocher2DInterpolator(np.array([x,y]).T, z)
Now you can call this interpolator with 2 coordinates to give you the corresponding interpolated data point:
>>> interpolator(x[10], y[10]) == z[10]
True
>>> interpolator(2, 300)
array(77.81343)
Note that you'll have to stay inside the convex hull of the input points, otherwise you'll get nan (or whatever is passed as the fill_value keyword to the interpolator):
>>> interpolator(2, 30)
array(nan)
Extrapolation is usually meaningless anyway, and your input points are scattered in a bit erratic way:
So even if extrapolation was possible I wouldn't believe it.
Just to demonstrate how the resulting interpolator is constrained to the convex hull of the input points, here's a surface plot of your data on a gridded mesh we create just for plotting:
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
# go linearly in the x grid
xline = np.linspace(min(x), max(x), 30)
# go logarithmically in the y grid (considering y distribution)
yline = np.logspace(np.log10(min(y)), np.log10(max(y)), 30)
# construct 2d grid from these
xgrid,ygrid = np.meshgrid(xline, yline)
# interpolate z data; same shape as xgrid and ygrid
z_interp = interpolator(xgrid, ygrid)
# create 3d Axes and plot surface and base points
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(xgrid, ygrid, z_interp, cmap='viridis',
vmin=min(z), vmax=max(z))
ax.plot(x, y, z, 'ro')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
plt.show()
Here's the output from two angles (it's better to rotate around interactively; such stills don't do the 3d representation justice):
There are two main features to note:
The surface nicely fits the red points, which is expected from interpolation. Fortunately the input points are nice and smooth so everything goes well with interpolation. (The fact that the red points are usually hidden by the surface is only due to how pyplot's renderer mishandles the relative position of complex 3d objects)
The surface is cut (due to nan values) along the convex hull of the input points, so even though our gridded arrays define a rectangular grid we only get a cut of the surface where interpolation makes sense.

Python: Linregress slope and y-intercept

I'm working on a program that can calculate the slope using the linregress native scipyy function, but I'm getting two errors (depending on how I try to fix it). The two lists should be two-dimensional, basically x and y values.
from __future__ import division
from scipy.stats import linregress
import matplotlib.pyplot as mplot
import numpy as np
xs=[[20.0, 80.0, 45.0, 42.0, 93.0, 98.0, 65.0, 43.0, 72.0, 36.0, 9.0, 60.0, 47.0, 84.0, 31.0, 46.0, 57.0, 76.0, 27.0, 85.0, 0.0, 39.0, 2.0, 56.0, 68.0, 6.0, 41.0, 28.0, 61.0, 12.0, 32.0, 1.0, 54.0, 77.0, 18.0, 86.0, 62.0, 23.0, 30.0, 69.0, 4.0, 71.0, 64.0, 92.0, 24.0, 79.0, 8.0, 35.0, 49.0, 53.0, 7.0, 59.0, 70.0, 37.0, 13.0, 15.0, 73.0, 89.0, 96.0, 83.0, 22.0, 95.0, 19.0, 67.0, 5.0, 88.0, 38.0, 50.0, 55.0, 52.0, 81.0, 58.0, 11.0, 51.0, 99.0, 78.0, 25.0, 33.0, 40.0, 75.0, 3.0, 91.0, 48.0, 90.0, 82.0, 26.0, 10.0, 16.0, 21.0, 66.0, 14.0, 87.0, 74.0, 97.0, 94.0, 44.0, 29.0, 17.0, 63.0, 34.0], [87.0, 17.0, 69.0, 72.0, 76.0, 62.0, 20.0, 77.0, 5.0, 49.0, 81.0, 3.0, 24.0, 36.0, 44.0, 91.0, 99.0, 35.0, 43.0, 50.0, 12.0, 54.0, 46.0, 30.0, 37.0, 45.0, 90.0, 85.0, 70.0, 83.0, 38.0, 22.0, 23.0, 0.0, 60.0, 47.0, 26.0, 1.0, 95.0, 73.0, 65.0, 94.0, 84.0, 8.0, 34.0, 56.0, 66.0, 13.0, 75.0, 52.0, 19.0, 55.0, 67.0, 39.0, 21.0, 80.0, 98.0, 33.0, 11.0, 68.0, 40.0, 32.0, 2.0, 79.0, 82.0, 93.0, 96.0, 88.0, 14.0, 92.0, 41.0, 89.0, 28.0, 29.0, 42.0, 6.0, 86.0, 74.0, 58.0, 16.0, 31.0, 64.0, 15.0, 53.0, 25.0, 59.0, 61.0, 78.0, 51.0, 7.0, 57.0, 9.0, 97.0, 63.0, 48.0, 71.0, 18.0, 10.0, 4.0, 27.0]]
ys=[[155.506, 50.592, 104.447, 111.318, 36.148, 36.87, 74.266, 106.413, 58.341, 122.563, 180.555, 85.202, 96.84, 50.726, 126.56, 100.686, 88.303, 54.797, 138.487, 44.946, 200.9, 116.524, 193.652, 82.8, 65.823, 184.436, 113.738, 133.458, 83.765, 167.408, 129.491, 200.469, 89.238, 51.799, 159.217, 49.382, 78.443, 146.051, 129.045, 63.805, 185.564, 65.614, 74.243, 43.408, 140.863, 53.446, 182.767, 127.373, 94.494, 91.079, 187.194, 81.254, 68.702, 121.368, 164.756, 169.696, 59.483, 45.978, 33.057, 47.12, 154.755, 33.872, 160.754, 70.256, 190.393, 38.398, 113.188, 100.493, 84.511, 88.635, 49.353, 81.821, 178.876, 95.307, 32.2, 54.715, 141.389, 132.337, 109.673, 57.611, 189.251, 39.283, 97.31, 41.173, 47.529, 140.03, 173.058, 160.288, 154.773, 67.903, 164.718, 42.032, 60.739, 28.656, 34.302, 107.022, 137.344, 160.195, 73.636, 123.797], [14.138, 100.87, 30.287, 28.675, 21.826, 42.445, 97.938, 29.574, 125.976, 59.404, 26.609, 125.743, 95.329, 75.467, 59.497, 15.342, 9.834, 77.402, 65.019, 54.468, 112.64, 45.466, 55.197, 79.992, 71.146, 55.39, 14.795, 15.971, 28.535, 25.862, 73.239, 92.455, 87.635, 137.6, 38.59, 53.718, 86.26, 130.567, 11.274, 33.867, 40.035, 11.07, 16.109, 114.732, 76.552, 45.85, 31.827, 110.877, 26.292, 55.738, 101.801, 48.601, 33.632, 66.647, 98.39, 23.904, 11.172, 78.215, 109.417, 31.653, 68.368, 79.593, 124.548, 21.513, 19.828, 13.48, 9.993, 22.043, 108.229, 16.904, 66.704, 12.262, 79.947, 85.012, 66.754, 124.114, 17.548, 25.872, 45.392, 101.775, 78.085, 36.358, 101.795, 52.045, 87.637, 42.784, 37.011, 26.036, 50.146, 119.666, 42.514, 113.313, 9.125, 42.394, 51.954, 26.898, 96.678, 112.108, 125.252, 86.296]]
slope, intercept, r_value, std_err = linregress(xs,ys)
print(slope)
My error is:
in linregress
ssxm, ssxym, ssyxm, ssym = np.cov(x, y, bias=1).flat
ValueError: too many values to unpack (expected 4)
I've tried changing my code to something like this:
slope, intercept, r_value, std_err = linregress(xs[:,0], ys[:,0])
But then my error becomes a TypeError:
TypeError: list indices must be integers or slices, not tuple
Does anyone have any suggestions? Perhaps there's something I don't understand about the use of the linregress function. I'm sure my first error has to do with my lists being 2D. For the second error, I'm lost.

You have two problems:
When interpreted as arrays, your variables xs and ys are two-dimensional with shape (2, 100). When linregress is given both arguments x and y, it expects them to be one-dimensional arrays.
As you can see in the "Returns" section of the docstring, linregress returns five values, not four.
You'll have to call linregress twice, and handle the five return values. For example,
In [144]: slope, intercept, rvalue, pvalue, stderr = linregress(xs[0], ys[0])
In [145]: slope, intercept, rvalue
Out[145]: (-1.7059670627062702, 187.5658196039604, -0.9912859597363385)
In [146]: slope, intercept, rvalue, pvalue, stderr = linregress(xs[1], ys[1])
In [147]: slope, intercept, rvalue
Out[147]: (-1.2455432103210327, 121.51968891089112, -0.9871123119133126)

Categories

Resources