Quantile Regression with Tensorflow Probability - python

I am trying to give tensorflow probability a try. I have a simple quantile regression in R. I would like to get the same results from tensorflow probability.
R Quantile Regression
library("quantreg")
mtcars_url <- "https://gist.githubusercontent.com/seankross/a412dfbd88b3db70b74b/raw/5f23f993cd87c283ce766e7ac6b329ee7cc2e1d1/mtcars.csv"
mtcars <- readr::read_csv(mtcars_url)
rqfit <- rq(mpg ~ wt, data = mtcars, tau = seq(.1, .9, by = 0.1))
predict(rqfit, mtcars, interval = c("confidence"),
level = .95)
The output for the above is 9 quantile predictions for each value
tau= 0.1 tau= 0.2 tau= 0.3 tau= 0.4 tau= 0.5 tau= 0.6 tau= 0.7 tau= 0.8 tau= 0.9
1 20.493299 20.92800 21.238356 22.176471 22.338816 23.592283 24.475462 25.12302 27.97207
2 18.837113 19.33680 19.910959 20.938971 21.181250 22.313183 23.110731 23.78409 26.37404
3 22.441753 22.80000 22.800000 23.632353 23.700658 25.097106 26.081028 26.69824 29.85210
4 16.628866 17.21520 18.141096 19.288971 19.637829 20.607717 21.291089 21.99884 24.24333
5 15.167526 15.81120 16.969863 18.197059 18.616447 19.479100 20.086915 20.81743 22.83330
6 15.037629 15.68640 16.865753 18.100000 18.525658 19.378778 19.979877 20.71242 22.70797
7 14.323196 15.00000 16.293151 17.566176 18.026316 18.827010 19.391169 20.13484 22.01862
8 16.791237 17.37120 18.271233 19.410294 19.751316 20.733119 21.424886 22.13011 24.40000
9 17.051031 17.62080 18.479452 19.604412 19.932895 20.933762 21.638962 22.34014 24.65067
10 15.167526 15.81120 16.969863 18.197059 18.616447 19.479100 20.086915 20.81743 22.83330
11 15.167526 15.81120 16.969863 18.197059 18.616447 19.479100 20.086915 20.81743 22.83330
12 11.075773 11.88000 13.690411 15.139706 15.756579 16.318971 16.715226 17.50948 18.88523
13 13.284021 14.00160 15.460274 16.789706 17.300000 18.024437 18.534868 19.29472 21.01594
14 12.959278 13.68960 15.200000 16.547059 17.073026 17.773633 18.267273 19.03219 20.70260
15 3.411856 4.51680 7.547945 9.413235 10.400000 10.400000 10.400000 11.31363 11.49042
16 2.281753 3.43104 6.642192 8.568824 9.610132 9.527203 9.468772 10.40000 10.40000
17 2.794845 3.92400 7.053425 8.952206 9.968750 9.923473 9.891571 10.81481 10.89508
18 23.221134 23.54880 23.424658 24.214706 24.245395 25.699035 26.723254 27.32833 30.60412
19 27.020619 27.19920 26.469863 27.053676 26.900987 28.633441 29.854108 30.40000 34.27019
20 25.591753 25.82640 25.324658 25.986029 25.902303 27.529904 28.676693 29.24484 32.89150
21 21.500000 21.89520 22.045205 22.928676 23.042434 24.369775 25.305004 25.93689 28.94342
22 14.647938 15.31200 16.553425 17.808824 18.253289 19.077814 19.658764 20.39737 22.33196
23 15.200000 15.84240 16.995890 18.221324 18.639145 19.504180 20.113674 20.84369 22.86464
24 12.569588 13.31520 14.887671 16.255882 16.800658 17.472669 17.946160 18.71714 20.32659
25 12.537113 13.28400 14.861644 16.231618 16.777961 17.447588 17.919401 18.69089 20.29526
26 24.942268 25.20240 24.804110 25.500735 25.448355 27.028296 28.141504 28.71977 32.26482
27 23.610825 23.92320 23.736986 24.505882 24.517763 26.000000 27.044367 27.64337 30.98013
28 27.683093 27.83568 27.000822 27.548676 27.364013 29.145080 30.400000 30.93557 34.90940
29 16.921134 17.49600 18.375342 19.507353 19.842105 20.833441 21.531924 22.23513 24.52534
30 19.519072 19.99200 20.457534 21.448529 21.657895 22.839871 23.672679 24.33542 27.03205
31 14.323196 15.00000 16.293151 17.566176 18.026316 18.827010 19.391169 20.13484 22.01862
32 19.454124 19.92960 20.405479 21.400000 21.612500 22.789711 23.619160 24.28291 26.96938
Tensorflow Probability Try
# pip3 install tensorflow_probability must be installed
from pprint import pprint
import numpy as np
import pandas as pd
import tensorflow.compat.v2 as tf
import tensorflow_probability as tfp
import termplotlib as tpl
np.set_printoptions(formatter={'float': lambda x: "{0:0.1f}".format(x)})
tf.enable_v2_behavior()
tfd = tfp.distributions
mtcars_url = "https://gist.githubusercontent.com/seankross/a412dfbd88b3db70b74b/raw/5f23f993cd87c283ce766e7ac6b329ee7cc2e1d1/mtcars.csv"
df_cars = pd.read_csv(mtcars_url)
y = df_cars.mpg
x = df_cars.wt
y = np.array(y)
x = np.array(x)
negloglik = lambda y, rv_y: -rv_y.log_prob(y)
model = tf.keras.Sequential([
tf.keras.layers.Dense(1 + 1),
tfp.layers.DistributionLambda(
lambda t: tfd.Normal(loc=t[..., :1],
scale=1e-3 + tf.math.softplus(0.05 * t[...,1:]))),
])
# Do inference.
model.compile(optimizer=tf.optimizers.Adam(learning_rate=0.01), loss=negloglik)
model.fit(x, y, epochs=1000, verbose=False);
# generate 40000 samples
n = 40000
yhat = model(np.array(x))
a = yhat.sample(n).numpy().reshape(32,n,1)
# check that the quantiles of the 40000 samples are similar to R script
quantiles = np.linspace(start=0.1, stop=0.9, num=9)
array_quants = np.quantile(a, q = quantiles, axis=1)
print(array_quants.reshape(32, 9))
The output here is:
[[13.2 13.3 13.2 13.2 13.2 13.2 13.2 13.2 13.2]
[13.2 13.2 13.2 13.2 13.2 13.2 13.2 13.2 13.2]
[13.2 13.2 13.2 13.2 13.2 13.2 13.2 13.2 13.2]
[13.2 13.2 13.2 13.2 13.2 15.1 15.1 15.1 15.2]
[15.1 15.1 15.2 15.2 15.2 15.2 15.1 15.2 15.2]
[15.2 15.2 15.2 15.2 15.1 15.2 15.2 15.2 15.1]
[15.2 15.2 15.2 15.2 15.2 15.2 15.1 15.1 15.2]
[15.1 16.9 16.9 16.9 16.9 16.9 16.9 16.9 16.9]
[16.9 16.9 16.9 16.9 16.9 16.9 16.9 16.9 16.9]
[16.9 16.9 16.9 16.9 16.9 16.9 16.9 16.9 16.9]
[16.9 16.9 16.9 16.9 16.9 16.9 18.3 18.3 18.3]
[18.3 18.3 18.3 18.3 18.3 18.3 18.3 18.3 18.3]
[18.3 18.3 18.3 18.3 18.3 18.3 18.3 18.3 18.3]
[18.3 18.3 18.3 18.3 18.3 18.3 18.3 18.3 18.3]
[18.3 18.3 19.4 19.3 19.4 19.4 19.4 19.4 19.3]
[19.4 19.4 19.3 19.4 19.3 19.4 19.4 19.4 19.3]
[19.4 19.4 19.4 19.3 19.4 19.4 19.4 19.3 19.4]
[19.4 19.3 19.4 19.4 19.4 19.4 19.3 20.2 20.3]
[20.2 20.2 20.2 20.3 20.2 20.2 20.2 20.2 20.2]
[20.3 20.2 20.2 20.3 20.2 20.3 20.3 20.3 20.2]
[20.3 20.3 20.3 20.2 20.2 20.3 20.2 20.3 20.3]
[20.3 20.3 20.2 21.1 21.1 21.1 21.1 21.1 21.2]
[21.1 21.1 21.1 21.1 21.1 21.1 21.1 21.1 21.1]
[21.1 21.2 21.2 21.1 21.1 21.1 21.1 21.2 21.1]
[21.1 21.2 21.2 21.1 21.1 21.1 21.2 21.1 22.2]
[22.2 22.2 22.2 22.2 22.2 22.2 22.2 22.2 22.2]
[22.2 22.2 22.2 22.2 22.2 22.2 22.2 22.2 22.2]
[22.2 22.2 22.2 22.2 22.2 22.2 22.2 22.3 22.2]
[22.2 22.2 22.2 22.2 24.9 24.9 24.9 24.9 24.9]
[24.8 24.9 24.9 24.9 24.9 24.8 24.9 24.8 24.9]
[24.9 24.9 24.9 24.9 24.9 24.9 24.9 24.9 24.9]
[24.9 24.9 24.9 24.9 24.9 24.8 24.9 24.8 24.9]]
I am still learning TFP so I am sure there is some simple mistake I made in the model specification, but it is not currently obvious to me what that mistake is.


Try to adjust the learning rate and see if the model converges correctly. Also, with the reshape function in your code, you probably intended to do a transpose. With a learning rate of 1.0 and some code fixes, this is what I got:
>>> n = 1000
>>> a = yhat.sample(n)[:,:,0].numpy().T
>>> quantiles = np.linspace(start=0.1, stop=0.9, num=9)
>>> array_quants = np.quantile(a, q = quantiles, axis=1)
>>> np.round(array_quants.T,1)
array([[18.1, 19.8, 21.1, 22. , 23.1, 24.2, 25.4, 26.5, 28. ],
[16.6, 18.4, 19.8, 21. , 21.8, 23. , 24.3, 25.7, 27.4],
[19.9, 21.5, 22.9, 23.8, 24.7, 25.6, 26.6, 28. , 29.4],
[14.2, 16.1, 17.7, 19.1, 20.3, 21.5, 22.7, 24.4, 26.4],
[12.8, 15. , 16.4, 17.7, 19. , 20.3, 21.8, 23.4, 25.4],
[12.3, 14.6, 16.3, 17.7, 19. , 20.2, 21.7, 23.2, 25.2],
[12.1, 14.5, 16.2, 17.3, 18.6, 19.7, 21.3, 23. , 25.1],
[14.6, 16.3, 17.5, 19.1, 20.4, 21.6, 22.9, 24.2, 26.6],
[14.9, 17. , 18.4, 19.5, 20.5, 21.7, 22.9, 24.4, 26.5],
[13.1, 15.2, 17. , 18.2, 19.3, 20.6, 21.9, 23.4, 25.4],
[12.9, 15. , 16.7, 18.2, 19.4, 20.7, 22. , 23.6, 25.7],
[ 8.7, 11.6, 13.3, 14.5, 16.1, 17.5, 19. , 20.8, 23. ],
[10.7, 13.3, 15.1, 16.5, 17.7, 19.1, 20.6, 22.3, 24.3],
[11.1, 13.3, 14.9, 16.3, 17.4, 18.9, 20.4, 21.9, 24.1],
[ 2. , 5. , 7.1, 8.6, 10.2, 11.9, 13.4, 15.9, 18.6],
[-0.1, 3.9, 6.1, 8. , 9.7, 11.6, 13.5, 15.5, 17.9],
[ 0.3, 3.9, 6.3, 8.1, 9.8, 11.5, 13.1, 15.4, 18.4],
[20.6, 22.3, 23.3, 24.3, 25.2, 26.2, 27.2, 28.3, 30. ],
[24.2, 25.6, 26.6, 27.6, 28.3, 29.2, 30. , 30.8, 32. ],
[22.7, 24.1, 25.1, 26.1, 26.9, 27.9, 28.8, 29.9, 31.3],
[19. , 20.9, 22. , 23.3, 24. , 25. , 26. , 27.4, 29.1],
[12.4, 14.6, 16.3, 17.7, 18.9, 20. , 21.5, 23.1, 25.7],
[13.2, 15.3, 16.8, 18.2, 19.2, 20.5, 22. , 23.4, 25.4],
[10.5, 13.1, 14.9, 16.2, 17.4, 18.6, 20.1, 21.9, 24.3],
[10.3, 12.7, 14.4, 15.8, 17.4, 18.7, 20. , 21.9, 24.3],
[22.4, 23.9, 24.9, 25.8, 26.6, 27.5, 28.3, 29.3, 30.8],
[21.1, 22.6, 23.6, 24.5, 25.5, 26.3, 27.2, 28. , 29.6],
[25.1, 26.3, 27.2, 28. , 28.7, 29.4, 30.1, 31. , 32.3],
[14.4, 16.3, 17.7, 18.9, 20. , 21.3, 22.7, 24.2, 26.5],
[16.9, 18.8, 20.2, 21.2, 22.5, 23.5, 24.6, 25.8, 27.5],
[12.4, 14.7, 16.1, 17.5, 18.7, 20. , 21.5, 23.1, 25.3],
[16.8, 18.7, 20. , 21.2, 22.5, 23.5, 24.6, 26.3, 28.1]])

Related

matplotlib.pyplot fill_between shape error

I am trying to display a band using an upper and lower plots. but I keep getting an error. I am using matplotlib.pyplot fill_between.
Error Message:
ValueError: x and y must have same first dimension, but have shapes (1,) and (448, 1)
plt.xlim([0, 45.5])
plt.ylim([-16.50, 15.000])
plt.plot(len(X)-1, y_predicted, 'b', linewidth=1)
plt.fill_between(X, y_predit_lower, y_predit_upper, alpha=0.1, color='green')
plt.plot(X, y_predit_upper, 'g', linewidth=1, linestyle='--')
plt.plot(X, y_predit_lower, 'g', linewidth=1, linestyle='--')
plt.axhline(y=10.70, color='r', linestyle='-.',linewidth=2)
plt.xticks(np.arange(0, 45.5, 1.0), fontsize=8)
plt.yticks(np.arange(-16.50, 15.00, 0.50), fontsize=8)
plt.title("Pressure Gradient Valve Size (27mm)")
plt.xlabel("Time (sec)")
plt.ylabel("Pressure (mmHg)")
plt.grid()
plt.show()
For my x am using the values from a column of a DataFrame:
X = df_train['Time'].to_numpy('float')
This is the line of the code thats gives me the error:
plt.fill_between(X, y_predit_lower, y_predit_upper, alpha=0.1, color='green')
Error Message I get:
ValueError: x and y must have same first dimension, but have shapes (1,) and (448, 1)
In: print(X.shape)
Out: (448,)
In: print(y_predit_upper.shape)
Out: (448,)
In: print(y_predit_lower.shape)
Out: (448,)
In: print(X)
Out:
[ 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1. 1.1 1.2 1.3 1.4
1.5 1.6 1.7 1.8 1.9 2. 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8
2.9 3. 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 4. 4.1 4.2
4.3 4.4 4.5 4.6 4.7 4.8 4.9 5. 5.1 5.2 5.3 5.4 5.5 5.6
5.7 5.8 5.9 6. 6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8 6.9 7.
7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8 7.9 8. 8.1 8.2 8.3 8.4
8.5 8.6 8.7 8.8 8.9 9. 9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8
9.9 10. 10.1 10.2 10.3 10.4 10.5 10.6 10.7 10.8 10.9 11. 11.1 11.2
11.3 11.4 11.5 11.6 11.7 11.8 11.9 12. 12.1 12.2 12.3 12.4 12.5 12.6
12.7 12.8 12.9 13. 13.1 13.2 13.3 13.4 13.5 13.6 13.7 13.8 13.9 14.
14.1 14.2 14.3 14.4 14.5 14.6 14.7 14.8 14.9 15. 15.1 15.2 15.3 15.4
15.5 15.6 15.7 15.8 15.9 16. 16.1 16.2 16.3 16.4 16.5 16.6 16.7 16.8
16.9 17. 17.1 17.2 17.3 17.4 17.5 17.6 17.7 17.8 17.9 18. 18.1 18.2
18.3 18.4 18.5 18.6 18.7 18.8 18.9 19. 19.1 19.2 19.3 19.4 19.5 19.6
19.7 19.8 19.9 20. 20.1 20.2 20.3 20.4 20.5 20.6 20.7 20.8 20.9 21.
21.1 21.2 21.3 21.4 21.5 21.6 21.7 21.8 21.9 22. 22.1 22.2 22.3 22.4
22.5 22.6 22.7 22.8 22.9 23. 23.1 23.2 23.3 23.4 23.5 23.6 23.7 23.8
23.9 24. 24.1 24.2 24.3 24.4 24.5 24.6 24.7 24.8 24.9 25. 25.1 25.2
25.3 25.4 25.5 25.6 25.7 25.8 25.9 26. 26.1 26.2 26.3 26.4 26.5 26.6
26.7 26.8 26.9 27. 27.1 27.2 27.3 27.4 27.5 27.6 27.7 27.8 27.9 28.
28.1 28.2 28.3 28.4 28.5 28.6 28.7 28.8 28.9 29. 29.1 29.2 29.3 29.4
29.5 29.6 29.7 29.8 29.9 30. 30.1 30.2 30.3 30.4 30.5 30.6 30.7 30.8
30.9 31. 31.1 31.2 31.3 31.4 31.5 31.6 31.7 31.8 31.9 32. 32.1 32.2
32.3 32.4 32.5 32.6 32.7 32.8 32.9 33. 33.1 33.2 33.3 33.4 33.5 33.6
33.7 33.8 33.9 34. 34.1 34.2 34.3 34.4 34.5 34.6 34.7 34.8 34.9 35.
35.1 35.2 35.3 35.4 35.5 35.6 35.7 35.8 35.9 36. 36.1 36.2 36.3 36.4
36.5 36.6 36.7 36.8 36.9 37. 37.1 37.2 37.3 37.4 37.5 37.6 37.7 37.8
37.9 38. 38.1 38.2 38.3 38.4 38.5 38.6 38.7 38.8 38.9 39. 39.1 39.2
39.3 39.4 39.5 39.6 39.7 39.8 39.9 40. 40.1 40.2 40.3 40.4 40.5 40.6
40.7 40.8 40.9 41. 41.1 41.2 41.3 41.4 41.5 41.6 41.7 41.8 41.9 42.
42.1 42.2 42.3 42.4 42.5 42.6 42.7 42.8 42.9 43. 43.1 43.2 43.3 43.4
43.5 43.6 43.7 43.8 43.9 44. 44.1 44.2 44.3 44.4 44.5 44.6 44.7 44.8]
In: print(y_predit_upper.shape)
Out:
[-10.920185 -10.730879 -10.395649 -9.781197 -8.639384
-6.6007776 -3.5282364 -0.28529644 2.1445403 3.7071989
4.679699 5.2941465 5.695036 5.9663973 6.157011
6.2958107 6.400445 6.4820027 6.5476484 6.6021276
6.648663 6.6894903 6.72619 6.7599044 6.7914734
6.821528 6.8505487 6.8789124 6.906917 6.934807
6.962783 6.991015 7.01965 7.0488157 7.0786242
7.1091766 7.140561 7.1728573 7.2061357 7.2404547
7.275866 7.31241 7.3501153 7.389001 7.4290714
7.4703183 7.512725 7.5562544 7.600858 7.646476
7.6930337 7.740444 7.7886114 7.8374276 7.886779
7.9365463 7.986604 8.036831 8.087101 8.137291
8.187288 8.236979 8.286261 8.335035 8.383221
8.430737 8.477519 8.52351 8.568661 8.61293
8.65629 8.698717 8.740196 8.780713 8.82027
8.858864 8.8965 8.933188 8.96894 9.003771
9.037693 9.070728 9.102894 9.134211 9.1647
9.19438 9.223274 9.251405 9.2787895 9.305452
9.331414 9.356693 9.381311 9.405285 9.428637
9.451382 9.47354 9.495129 9.516164 9.536662
9.556641 9.576112 9.595093 9.613596 9.631638
9.649229 9.666382 9.683111 9.69943 9.715345
9.730873 9.746021 9.760802 9.775225 9.7893
9.803036 9.8164425 9.829529 9.842304 9.854776
9.866953 9.878841 9.890451 9.901789 9.912861
9.923676 9.9342375 9.944555 9.954636 9.964481
9.974102 9.983503 9.992689 10.001665 10.010438
10.019011 10.0273905 10.03558 10.043587 10.051413
10.059064 10.066545 10.07386 10.081012 10.088005
10.094845 10.101532 10.108074 10.114471 10.120729
10.126851 10.132839 10.138698 10.144428 10.150038
10.155523 10.1608925 10.166145 10.171288 10.176319
10.181244 10.186064 10.190783 10.1954 10.199922
10.204349 10.20868 10.212924 10.217076 10.221144
10.225128 10.229028 10.232847 10.236588 10.2402525
10.243841 10.247356 10.2508 10.254173 10.257479
10.2607155 10.263888 10.266995 10.270041 10.273026
10.275951 10.278817 10.281626 10.284379 10.287078
10.289722 10.292316 10.294858 10.297348 10.299793
10.302189 10.304538 10.30684 10.309098 10.311312
10.313485 10.315615 10.317703 10.319752 10.321762
10.323734 10.325667 10.327566 10.329426 10.331253
10.333044 10.334803 10.336527 10.338222 10.339883
10.341513 10.343113 10.344685 10.346227 10.34774
10.349226 10.350685 10.352116 10.353521 10.354902
10.356257 10.357589 10.358896 10.36018 10.36144
10.362679 10.363894 10.365089 10.366262 10.367414
10.368546 10.369658 10.370752 10.371826 10.372882
10.373919 10.374937 10.375938 10.376922 10.37789
10.3788395 10.379774 10.380693 10.381596 10.3824835
10.383356 10.384214 10.385057 10.385887 10.3867035
10.387505 10.388294 10.3890705 10.3898325 10.390584
10.391323 10.39205 10.392763 10.393466 10.394157
10.394838 10.395506 10.396166 10.396815 10.397452
10.398081 10.398699 10.399307 10.399906 10.400494
10.401075 10.401647 10.402208 10.402762 10.403307
10.403845 10.404373 10.404894 10.405405 10.4059105
10.406408 10.4068985 10.40738 10.407856 10.408323
10.408785 10.409239 10.409686 10.410128 10.410563
10.410991 10.411414 10.411829 10.412239 10.412644
10.413042 10.413435 10.413822 10.414204 10.414579
10.414951 10.415317 10.415678 10.416034 10.416384
10.416731 10.417071 10.417408 10.41774 10.418068
10.418391 10.41871 10.4190235 10.419333 10.41964
10.419942 10.42024 10.420534 10.420824 10.42111
10.421393 10.421673 10.421947 10.422218 10.422487
10.422752 10.423013 10.423271 10.423527 10.423778
10.4240265 10.424272 10.424515 10.424753 10.42499
10.425224 10.425454 10.425682 10.425907 10.426128
10.426348 10.426565 10.426779 10.4269905 10.4272
10.427407 10.427612 10.427814 10.428013 10.428211
10.428406 10.428598 10.428788 10.428976 10.429163
10.429345 10.429527 10.4297085 10.429886 10.430061
10.430235 10.430407 10.430576 10.430744 10.43091
10.431075 10.431237 10.431398 10.431557 10.431714
10.4318695 10.432024 10.432177 10.432327 10.432476
10.432623 10.43277 10.432913 10.433057 10.433199
10.433338 10.433476 10.433613 10.433748 10.433882
10.434015 10.434147 10.434278 10.434406 10.434532
10.43466 10.434786 10.434908 10.43503 10.435152
10.435272 10.43539 10.435509 10.435625 10.435741
10.435854 10.435968 10.43608 10.436192 10.436301
10.43641 10.436517 10.436625 10.43673 10.436835
10.436939 10.437042 10.437143 10.437244 10.4373455
10.437443 10.437542 10.437639 10.437736 10.437831
10.437925 10.43802 10.438112 10.438204 10.438295
10.438386 10.438476 10.438565 10.438652 10.438741
10.438827 10.438912 10.438997 10.438997 10.438997
10.438997 10.438997 10.438997 10.438997 10.438997
10.438997 10.438997 10.438997 ]
In: print(y_predit_lower.shape)
Out:
[-1.4231827e+01 -1.4231827e+01 -1.4231827e+01 -1.4231827e+01
-1.4231827e+01 -1.4231827e+01 -1.4231827e+01 -1.4231827e+01
-1.4231827e+01 -1.4231827e+01 -1.4231827e+01 -1.4229019e+01
-1.4225172e+01 -1.4219831e+01 -1.4212313e+01 -1.4201557e+01
-1.4185902e+01 -1.4162625e+01 -1.4127128e+01 -1.4071282e+01
-1.3980150e+01 -1.3825264e+01 -1.3550985e+01 -1.3048252e+01
-1.2114041e+01 -1.0446091e+01 -7.9321928e+00 -5.2788787e+00
-3.2908306e+00 -2.0122919e+00 -1.2166102e+00 -7.1388006e-01
-3.8587976e-01 -1.6385698e-01 -7.9002380e-03 1.0566306e-01
1.9127297e-01 2.5800228e-01 3.1171203e-01 3.5628605e-01
3.9436054e-01 4.2776465e-01 4.5779157e-01 4.8537636e-01
5.1120520e-01 5.3579545e-01 5.5953956e-01 5.8274651e-01
6.0565925e-01 6.2847805e-01 6.5136743e-01 6.7446661e-01
6.9789529e-01 7.2175813e-01 7.4614692e-01 7.7114415e-01
7.9682255e-01 8.2324672e-01 8.5047436e-01 8.7855363e-01
9.0752649e-01 9.3742609e-01 9.6827579e-01 1.0000916e+00
1.0328760e+00 1.0666242e+00 1.1013203e+00 1.1369352e+00
1.1734290e+00 1.2107525e+00 1.2488456e+00 1.2876358e+00
1.3270454e+00 1.3669858e+00 1.4073639e+00 1.4480829e+00
1.4890399e+00 1.5301342e+00 1.5712638e+00 1.6123290e+00
1.6532354e+00 1.6938915e+00 1.7342124e+00 1.7741199e+00
1.8135443e+00 1.8524213e+00 1.8906975e+00 1.9283261e+00
1.9652677e+00 2.0014882e+00 2.0369649e+00 2.0716777e+00
2.1056142e+00 2.1387653e+00 2.1711292e+00 2.2027068e+00
2.2335000e+00 2.2635174e+00 2.2927685e+00 2.3212667e+00
2.3490210e+00 2.3760500e+00 2.4023676e+00 2.4279904e+00
2.4529357e+00 2.4772196e+00 2.5008607e+00 2.5238762e+00
2.5462823e+00 2.5680971e+00 2.5893383e+00 2.6100211e+00
2.6301632e+00 2.6497784e+00 2.6688843e+00 2.6874938e+00
2.7056236e+00 2.7232871e+00 2.7404976e+00 2.7572689e+00
2.7736154e+00 2.7895460e+00 2.8050761e+00 2.8202147e+00
2.8349762e+00 2.8493690e+00 2.8634038e+00 2.8770909e+00
2.8904424e+00 2.9034648e+00 2.9161687e+00 2.9285626e+00
2.9406557e+00 2.9524565e+00 2.9639721e+00 2.9752111e+00
2.9861798e+00 2.9968877e+00 3.0073400e+00 3.0175438e+00
3.0275064e+00 3.0372334e+00 3.0467329e+00 3.0560083e+00
3.0650678e+00 3.0739164e+00 3.0825577e+00 3.0909996e+00
3.0992465e+00 3.1073031e+00 3.1151748e+00 3.1228657e+00
3.1303816e+00 3.1377258e+00 3.1449032e+00 3.1519175e+00
3.1587734e+00 3.1654744e+00 3.1720252e+00 3.1784282e+00
3.1846886e+00 3.1908092e+00 3.1967940e+00 3.2026453e+00
3.2083673e+00 3.2139630e+00 3.2194352e+00 3.2247872e+00
3.2300220e+00 3.2351422e+00 3.2401505e+00 3.2450500e+00
3.2498436e+00 3.2545323e+00 3.2591219e+00 3.2636104e+00
3.2680025e+00 3.2723012e+00 3.2765074e+00 3.2806249e+00
3.2846537e+00 3.2885976e+00 3.2924585e+00 3.2962365e+00
3.2999353e+00 3.3035574e+00 3.3071012e+00 3.3105736e+00
3.3139715e+00 3.3172994e+00 3.3205595e+00 3.3237495e+00
3.3268752e+00 3.3299356e+00 3.3329334e+00 3.3358698e+00
3.3387456e+00 3.3415632e+00 3.3443227e+00 3.3470273e+00
3.3496766e+00 3.3522720e+00 3.3548145e+00 3.3573065e+00
3.3597488e+00 3.3621411e+00 3.3644867e+00 3.3667846e+00
3.3690367e+00 3.3712454e+00 3.3734097e+00 3.3755307e+00
3.3776112e+00 3.3796487e+00 3.3816485e+00 3.3836088e+00
3.3855305e+00 3.3874140e+00 3.3892622e+00 3.3910737e+00
3.3928509e+00 3.3945937e+00 3.3963022e+00 3.3979788e+00
3.3996229e+00 3.4012361e+00 3.4028187e+00 3.4043717e+00
3.4058938e+00 3.4073887e+00 3.4088535e+00 3.4102931e+00
3.4117036e+00 3.4130898e+00 3.4144497e+00 3.4157834e+00
3.4170928e+00 3.4183784e+00 3.4196396e+00 3.4208779e+00
3.4220929e+00 3.4232869e+00 3.4244580e+00 3.4256082e+00
3.4267378e+00 3.4278469e+00 3.4289360e+00 3.4300056e+00
3.4310555e+00 3.4320869e+00 3.4331002e+00 3.4340954e+00
3.4350724e+00 3.4360318e+00 3.4369760e+00 3.4379010e+00
3.4388113e+00 3.4397054e+00 3.4405851e+00 3.4414482e+00
3.4422965e+00 3.4431300e+00 3.4439492e+00 3.4447536e+00
3.4455457e+00 3.4463229e+00 3.4470873e+00 3.4478393e+00
3.4485774e+00 3.4493046e+00 3.4500179e+00 3.4507203e+00
3.4514103e+00 3.4520888e+00 3.4527564e+00 3.4534125e+00
3.4540586e+00 3.4546938e+00 3.4553175e+00 3.4559317e+00
3.4565368e+00 3.4571309e+00 3.4577150e+00 3.4582901e+00
3.4588556e+00 3.4594121e+00 3.4599595e+00 3.4604993e+00
3.4610300e+00 3.4615517e+00 3.4620652e+00 3.4625711e+00
3.4630685e+00 3.4635587e+00 3.4640403e+00 3.4645157e+00
3.4649830e+00 3.4654431e+00 3.4658966e+00 3.4663415e+00
3.4667811e+00 3.4672141e+00 3.4676394e+00 3.4680586e+00
3.4684715e+00 3.4688792e+00 3.4692798e+00 3.4696741e+00
3.4700637e+00 3.4704461e+00 3.4708233e+00 3.4711957e+00
3.4715614e+00 3.4719224e+00 3.4722786e+00 3.4726286e+00
3.4729748e+00 3.4733148e+00 3.4736505e+00 3.4739819e+00
3.4743066e+00 3.4746289e+00 3.4749455e+00 3.4752569e+00
3.4755650e+00 3.4758687e+00 3.4761677e+00 3.4764628e+00
3.4767542e+00 3.4770417e+00 3.4773250e+00 3.4776039e+00
3.4778790e+00 3.4781504e+00 3.4784188e+00 3.4786835e+00
3.4789438e+00 3.4792008e+00 3.4794540e+00 3.4797049e+00
3.4799519e+00 3.4801960e+00 3.4804363e+00 3.4806743e+00
3.4809079e+00 3.4811401e+00 3.4813676e+00 3.4815922e+00
3.4818144e+00 3.4820347e+00 3.4822516e+00 3.4824648e+00
3.4826765e+00 3.4828849e+00 3.4830904e+00 3.4832940e+00
3.4834948e+00 3.4836936e+00 3.4838891e+00 3.4840827e+00
3.4842734e+00 3.4844618e+00 3.4846487e+00 3.4848323e+00
3.4850144e+00 3.4851933e+00 3.4853716e+00 3.4855466e+00
3.4857192e+00 3.4858909e+00 3.4860601e+00 3.4862275e+00
3.4863930e+00 3.4865561e+00 3.4867177e+00 3.4868770e+00
3.4870348e+00 3.4871902e+00 3.4873438e+00 3.4874964e+00
3.4876461e+00 3.4877954e+00 3.4879422e+00 3.4880881e+00
3.4882312e+00 3.4883738e+00 3.4885139e+00 3.4886527e+00
3.4887900e+00 3.4889264e+00 3.4890614e+00 3.4891939e+00
3.4893255e+00 3.4894552e+00 3.4895840e+00 3.4897108e+00
3.4898376e+00 3.4899626e+00 3.4900851e+00 3.4902072e+00
3.4903274e+00 3.4904480e+00 3.4905648e+00 3.4906826e+00
3.4907985e+00 3.4909120e+00 3.4910259e+00 3.4911375e+00
3.4912481e+00 3.4913583e+00 3.4914665e+00 3.4915743e+00
3.4916811e+00 3.4917865e+00 3.4918904e+00 3.4919934e+00
3.4920964e+00 3.4921975e+00 3.4922967e+00 3.4923968e+00
3.4924951e+00 3.4925923e+00 3.4926887e+00 3.4927831e+00
3.4928784e+00 3.4929719e+00 3.4930644e+00 3.4931564e+00
3.4932475e+00 3.4933367e+00 3.4934263e+00 3.4935131e+00
3.4936023e+00 3.4936886e+00 3.4937744e+00 3.4938593e+00
3.4939427e+00 3.4940267e+00 3.4941092e+00 3.4941907e+00
3.4942718e+00 3.4943528e+00 3.4944315e+00 3.4945107e+00
3.4945889e+00 3.4946666e+00 3.4947433e+00 3.4948187e+00]
Dont know what I am missing here with the data structure.

pandas: round up to closet float number defined by user

I have one array which contains continuous values. I need to round up those values to the closet float value. ex: 32.25 to 32.50 , 30.29 to 30.50, 33.75 to 34.00. In short: if it is from .1 to .49 round up to .50 and if it is from .51 to .99 round up to .00. How can I do it. Thank you in advance.
array([32.5 , 32.49, 32.48, 32.47, 32.46, 32.45, 32.44, 32.43, 32.42,
32.41, 32.4 , 32.39, 32.38, 32.37, 32.36, 32.35, 32.34, 32.33,
32.32, 32.31, 32.3 , 32.29, 32.28, 32.27, 32.26, 32.25, 15.75,
15.76, 15.77, 15.78, 15.79, 15.8 , 15.81, 15.82, 15.83, 15.84,
15.85, 15.86, 15.87, 15.88, 15.89, 15.9 , 15.91, 15.92, 15.93,
15.94, 15.95, 15.96, 15.97, 15.98, 15.99, 16. , 16.01, 16.02,
16.03, 16.04, 16.05, 16.06, 16.07, 16.08, 16.09, 16.1 , 16.11,
16.12, 16.13, 16.14, 16.15, 16.16, 16.17, 16.18, 16.19, 16.2 ,
16.21, 16.22, 16.23, 16.24, 16.25, 16.26, 16.27, 16.28, 16.29,
16.3 , 16.31, 16.32, 16.33, 16.34, 16.35, 16.36, 16.37, 16.38,
16.39, 16.4 , 16.41, 16.42, 16.43, 16.44, 16.45, 16.46, 16.47,
16.48, 16.49, 16.5 , 25.25, 25.5 , 25.51, 25.52, 25.53, 25.54,
25.55, 25.56, 25.57, 25.58, 25.59, 25.6 , 25.61, 25.62, 25.63,
25.64, 25.65, 25.66, 25.67, 25.68, 25.69, 25.7 , 25.71, 25.72,
25.73, 25.74, 26. , 26.01, 26.02, 26.03, 26.04, 26.05, 26.06,
26.07, 26.08, 26.09, 26.1 , 26.11, 26.12, 26.13, 26.14, 26.15,
26.16, 26.17, 26.18, 26.19, 26.2 , 26.21, 26.22, 26.23, 26.24,
26.25, 26.26, 26.27, 26.28, 26.29, 26.3 , 26.31, 26.32, 26.5 ,
26.49, 26.48, 26.47, 26.46, 26.45, 26.44, 26.43, 26.42, 26.41,
26.4 , 26.39, 26.38, 26.37, 26.36, 26.35, 26.34, 26.33, 28.5 ,
28.51, 28.52, 28.53, 28.54, 28.55, 28.56, 28.57, 28.58, 28.59,
28.6 , 28.61, 28.62, 28.63, 28.64, 28.65, 28.66, 30.5 , 30.49,
30.48, 30.47, 30.46, 30.45, 30.44, 30.43, 30.42, 30.41, 30.4 ,
30.39, 30.38, 30.37, 30.36, 30.35, 30.34, 30.33, 30.32, 30.31,
30.3 , 30.29, 30.28, 30.27, 30.26, 30.25])

Did you not experiment with this? numpy is built for experimentation.
array = (array * 2 + 0.4999).round() / 2

Another solution:
import math
[math.modf(item)[1] + 0.5 if (0.1 <= ( item % 1) <= 0.5) else math.modf(item)[1] for item in array]

use the below code
round_off_values = np.round_(array, decimals = 1)

Create new columns in pandas df by grouping and performing operations on an existing column

I have a dataframe that looks like this (Minimal Reproducible Example)
thermometers = ['T-10000_0001', 'T-10000_0002','T-10000_0003', 'T-10000_0004',
'T-10001_0001', 'T-10001_0002', 'T-10001_0003', 'T-10001_0004',
'T-10002_0001', 'T-10002_0003', 'T-10002_0003', 'T-10002_0004']
temperatures = [15.1, 14.9, 12.7, 10.8,
19.8, 18.3, 17.7, 18.1,
20.0, 16.4, 17.6, 19.3]
df_set = {'thermometers': thermometers,
'Temperatures': temperatures}
df = pd.DataFrame(df_set)
Index
Thermometer
Temperature
0
T-10000_0001
14.9
1
T-10000_0002
12.7
2
T-10000_0003
12.7
3
T-10000_0004
10.8
4
T-10001_0001
19.8
5
T-10001_0002
18.3
6
T-10001_0003
17.7
7
T-10001_0004
18.1
8
T-10002_0001
20.0
9
T-10002_0002
16.4
10
T-10002_0003
17.6
11
T-10002_0004
19.3
I am trying to group the thermometers (i.e 'T-10000', 'T-10001', 'T-10002'), and create new columns with the min, max and average of each thermometer reading. So my final data frame would look like this
Index
Thermometer
min_temp
average_temp
max_temp
0
T-10000
10.8
12.8
14.9
1
T-10001
17.7
18.5
19.8
2
T-10002
16.4
18.3
20.0
I tried creating a separate function which I think requires regular expression, but I'm unable to figure out how to go about it. Any help will be much appreciated.

Use groupby by splitting with your delimiter _. Then, just aggregate with whatever functions you need.
>>> df.groupby(df['thermometers']\
.str.split('_'). \
.str.get(0)).agg(['min', 'mean', 'max'])
min mean max
thermometers
T-10000 10.8 13.375 15.1
T-10001 17.7 18.475 19.8
T-10002 16.4 18.325 20.0

Another approach with str.extract to avoid the call to str.get:
(df['Temperatures']
.groupby(df['thermometers'].str.extract('(^[^_]+)', expand=False))
.agg(['min', 'mean'])
)
Output:
min mean
thermometers
T-10000 10.8 13.375
T-10001 17.7 18.475
T-10002 16.4 18.325

Read a matrix from a .txt file

How would I go about reading a file like:
4
1.66 3.79 2.65 4.25
4.46 1.46 8.51 3.14
3.51 1.44 8.26 7.87
0.40 3.31 1.97 2.30
1.54 2.84 9.07 8.44
Where the first number is the number of equations in the system of equations, and the final row is the what the corresponding equations equal. For example, this would be:
1.66 3.79 2.65 4.25 | 1.54
4.46 1.46 8.51 3.14 | 2.84
3.51 1.44 8.26 7.87 | 9.07
0.40 3.31 1.97 2.30 | 8.44
as a matrix. My question is in python, how do I go about translating that into something like this:
A = [[1.66, 3.79, 2.65, 4.25], [4.46, 1.46, 8.51, 3.14], [3.51, 1.44, 8.26, 7.87], [0.40, 3.31, 1.97, 2.30]]
B = [1.54, 2.84, 9.07, 8.44]

In [7]: a = []
In [8]: b = []
In [9]: with open("a.txt") as f:
...: data = f.readlines()
...: no_of_equations = int(data[0].strip())
...: for i in data[1:no_of_equations+1]:
...: a.append(list(map(float,i.split())))
...: b.append(list(map(float, data[no_of_equations+1].split())))
...:
In [10]: a
Out[10]:
[[1.66, 3.79, 2.65, 4.25],
[4.46, 1.46, 8.51, 3.14],
[3.51, 1.44, 8.26, 7.87],
[0.4, 3.31, 1.97, 2.3]]
In [11]: b
Out[11]: [[1.54, 2.84, 9.07, 8.44]]

How do I extract the lower/upper triangle of a non-square numpy array?

So basically I have an array subslope consisting of the following:
import numpy as np
import matplotlib.pyplot as plt
slp = 1501.66*(np.ones((1000,1000), dtype=np.float32))
vsub = 2000*(np.ones((1000,1000), dtype=np.float32))
slope = np.triu(slp)
slope2 = np.tril(vsub, k=-1)
subslope = slope + slope2
This is the visual representation:
You can see the diagonal separating two parts of the array, the upper part has values of 1501.66 and 2000 below the diagonal. However, when I change the dimensions such that the number of columns is significantly larger than the number of rows like this:
slp = 1501.66*(np.ones((1000,2000), dtype=np.float32))
vsub = 2000*(np.ones((1000,2000), dtype=np.float32))
We get the following:
What I want is the diagonal to run from the top corner of the array to the bottom corner like this:
How can I achieve this?

You could use np.indices to create a boolean mask:
import numpy as np
import matplotlib.pyplot as plt
shape = (1000, 2000)
i,j = np.indices(shape)
m = np.ceil(i <= j*shape[0]/shape[1]).astype(bool)
subslope = np.empty(shape, np.float32)
subslope[m] = 1501.66
subslope[~m] = 2000
plt.imshow(subslope)

To generate your array, with nR rows and nC columns, filled:
at the diagonal with valDiag,
over the diagonal with valUpper,
below the diagonal with valLower,
you can use the following function:
def genDiag(nR, nC, valUpper, valDiag, valLower):
slope = nC / nR
tbl = np.full((nR, nC), valDiag, dtype=float)
for r in range(nR):
tbl[r, 0 : int(round(slope * r, 0))] = valLower
tbl[r, int(round(slope * (r + 1), 0)) : nC] = valUpper
return tbl
To test it, on smaller numbers, run:
res = genDiag(8, 14, 15.1, 0, 20.2)
print(res)
The result is:
[[ 0. 0. 15.1 15.1 15.1 15.1 15.1 15.1 15.1 15.1 15.1 15.1 15.1 15.1]
[20.2 20.2 0. 0. 15.1 15.1 15.1 15.1 15.1 15.1 15.1 15.1 15.1 15.1]
[20.2 20.2 20.2 20.2 0. 15.1 15.1 15.1 15.1 15.1 15.1 15.1 15.1 15.1]
[20.2 20.2 20.2 20.2 20.2 0. 0. 15.1 15.1 15.1 15.1 15.1 15.1 15.1]
[20.2 20.2 20.2 20.2 20.2 20.2 20.2 0. 0. 15.1 15.1 15.1 15.1 15.1]
[20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 0. 15.1 15.1 15.1 15.1]
[20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 0. 0. 15.1 15.1]
[20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 20.2 0. 0. ]]
If you want this table without separate filling for the diagonal,
first make up your mind whether the diagonal elements are to be filled
with "upper" or "lower" value, then pass the selected value as valDiag.

Categories

Resources