I have the following dataframe:
data
Out[120]:
High Low Open Close Volume Adj Close
Date
2018-01-02 12.66 12.50 12.52 12.66 20773300.0 10.842077
2018-01-03 12.80 12.67 12.68 12.76 29765600.0 10.927719
2018-01-04 13.04 12.77 12.78 12.98 37478200.0 11.116128
2018-01-05 13.22 13.04 13.06 13.20 46121900.0 11.304538
2018-01-08 13.22 13.11 13.21 13.15 33828300.0 11.261715
... ... ... ... ... ...
2020-06-25 6.05 5.80 5.86 6.03 73612700.0 6.030000
2020-06-26 6.07 5.81 6.04 5.91 118435400.0 5.910000
2020-06-29 6.07 5.81 5.91 6.01 58208400.0 6.010000
2020-06-30 6.10 5.90 5.98 6.08 61909300.0 6.080000
2020-07-01 6.18 5.95 6.10 5.98 62333600.0 5.980000
[629 rows x 6 columns]
Some of the dates are missing in Dates Column. I know i can do this to get all the dates:
pd.date_range(start, end, freq ='D')
Out[121]:
DatetimeIndex(['2018-01-01', '2018-01-02', '2018-01-03', '2018-01-04',
'2018-01-05', '2018-01-06', '2018-01-07', '2018-01-08',
'2018-01-09', '2018-01-10',
...
'2020-06-23', '2020-06-24', '2020-06-25', '2020-06-26',
'2020-06-27', '2020-06-28', '2020-06-29', '2020-06-30',
'2020-07-01', '2020-07-02'],
dtype='datetime64[ns]', length=914, freq='D')
How can i compare all the dates with the index and just add the dates which are missing.
Use DataFrame.reindex, working also if need some custom start and end datimes:
df = df.reindex(pd.date_range(start, end, freq ='D'))
Or DataFrame.asfreq for add missing datetimes between existing data:
df = df.asfreq('d')
I have a df as follows
dates winter summer rest Final
2020-01-01 00:15:00 65.5 71.5 73.0 NaN
2020-01-01 00:30:00 62.6 69.0 70.1 NaN
2020-01-01 00:45:00 59.6 66.3 67.1 NaN
2020-01-01 01:00:00 57.0 63.5 64.5 NaN
2020-01-01 01:15:00 54.8 60.9 62.3 NaN
2020-01-01 01:30:00 53.1 58.6 60.6 NaN
2020-01-01 01:45:00 51.7 56.6 59.2 NaN
2020-01-01 02:00:00 50.5 55.1 57.9 NaN
2020-01-01 02:15:00 49.4 54.2 56.7 NaN
2020-01-01 02:30:00 48.5 53.7 55.6 NaN
2020-01-01 02:45:00 47.9 53.4 54.7 NaN
2020-01-01 03:00:00 47.7 53.3 54.2 NaN
2020-01-01 03:15:00 47.9 53.1 54.1 NaN
2020-01-01 03:30:00 48.7 53.2 54.6 NaN
2020-01-01 03:45:00 50.2 54.1 55.8 NaN
2020-01-01 04:00:00 52.3 56.1 57.9 NaN
2020-04-28 12:30:00 225.1 200.0 209.8 NaN
2020-04-28 12:45:00 215.7 193.8 201.9 NaN
2020-04-28 13:00:00 205.6 186.9 193.4 NaN
2020-04-28 13:15:00 195.7 179.9 185.0 NaN
2020-04-28 13:30:00 186.7 173.4 177.4 NaN
2020-04-28 13:45:00 179.2 168.1 170.9 NaN
2020-04-28 14:00:00 173.8 164.4 166.3 NaN
2020-04-28 14:15:00 171.0 163.0 163.9 NaN
2020-04-28 14:30:00 170.7 163.5 163.6 NaN
2020-12-31 21:15:00 88.5 90.2 89.2 NaN
2020-12-31 21:30:00 85.2 88.5 87.2 NaN
2020-12-31 21:45:00 82.1 86.3 85.0 NaN
2020-12-31 22:00:00 79.4 84.1 83.2 NaN
2020-12-31 22:15:00 77.6 82.4 82.1 NaN
2020-12-31 22:30:00 76.4 81.2 81.7 NaN
2020-12-31 22:45:00 75.6 80.3 81.6 NaN
2020-12-31 23:00:00 74.7 79.4 81.3 NaN
2020-12-31 23:15:00 73.7 78.4 80.6 NaN
2020-12-31 23:30:00 72.3 77.2 79.5 NaN
2020-12-31 23:45:00 70.5 75.7 77.9 NaN
2021-01-01 00:00:00 68.2 73.8 75.7 NaN
The dates column has dates starting from 2020-01-01 00:15:00 till 2021-01-01 00:00:00 split at every 15 mins.
I also have the following date range conditions:
Winter: 01.11 - 20.03
Summer: 15.05 - 14.09
Rest: 21.03 - 14.05 & 15.09 - 31.10
What I want to do is to create a new column named season that checks every date in the dates column and assigns winter if the date is in Winter range, summer if it is in Summer range and rest if it is the Rest range.
Then, based on the value in the season column, the Final column must be filled. If the value in season column is 'winter', then the values from winter column must be placed, if the value in season column is 'summer', then the values from summer column must be placed and so on.
How can this be done?
Idea is normalize datetimes for same year, then filter by Series.between and set new column by numpy.select:
d = pd.to_datetime(df['dates'].dt.strftime('%m-%d-2020'))
m1 = d.between('2020-11-01','2020-12-31') | d.between('2020-01-01','2020-03-20')
m2 = d.between('2020-05-15','2020-09-14')
df['Final'] = np.select([m1, m2], ['Winter','Summer'], default='Rest')
print (df)
dates winter summer rest Final
0 2020-01-01 00:15:00 65.5 71.5 73.0 Winter
1 2020-06-15 00:30:00 62.6 69.0 70.1 Summer
2 2020-12-25 00:45:00 59.6 66.3 67.1 Winter
3 2020-10-10 01:00:00 57.0 63.5 64.5 Rest
I have tow dataframes stockData and EPSdata as below.
stockData will be
Close
Date
2018-12-31 157.066376
2018-09-30 223.994431
2018-06-30 183.036682
2018-03-31 165.263504
2017-12-31 166.014908
2017-09-30 150.650375
2017-06-30 140.227097
2017-03-31 139.301605
2016-12-31 111.821404
2016-09-30 108.589523
2016-06-30 91.333252
2016-03-31 103.495514
2015-12-31 99.414101
2015-09-30 103.730179
......
and EPSData will be
Surprise_perc
Date
2018-11-01 4.30
2018-07-31 7.83
2018-05-01 1.49
2018-02-01 1.83
2017-11-02 10.70
2017-08-01 6.37
2017-05-02 3.96
2017-01-31 4.35
2016-10-25 0.60
2016-07-26 2.16
.......
how can i combine both these dataframes ?
The first datapoint in EPSData," 2018-11-01 4.30" shoud be added to Q4 in stockData ie, to the datapoint "2018-12-31 157.066376".
The output should look like
Close Surprise_perc
Date
2018-12-31 157.066376 4.30
2018-09-30 223.994431 7.83
2018-06-30 183.036682 1.49
2018-03-31 165.263504 1.83
2017-12-31 166.014908 10.70
2017-09-30 150.650375 6.37
2017-06-30 140.227097 3.96
2017-03-31 139.301605 4.35
2016-12-31 111.821404 0.60
2016-09-30 108.589523 2.16
.....
here both dataframes are indexed on dates.
Use merge_asof with DataFrame.sort_index:
df = (pd.merge_asof(stockData.sort_index(),
EPSData.sort_index(),
left_index=True,
right_index=True)
.sort_index(ascending=False))
print (df)
Close Surprise_perc
Date
2018-12-31 157.066376 4.30
2018-09-30 223.994431 7.83
2018-06-30 183.036682 1.49
2018-03-31 165.263504 1.83
2017-12-31 166.014908 10.70
2017-09-30 150.650375 6.37
2017-06-30 140.227097 3.96
2017-03-31 139.301605 4.35
2016-12-31 111.821404 0.60
2016-09-30 108.589523 2.16
2016-06-30 91.333252 NaN
2016-03-31 103.495514 NaN
2015-12-31 99.414101 NaN
2015-09-30 103.730179 NaN
Similarly to this question, I'm interested in creating time series spirals. The solution doesn't necessarily have to be implemented in R or using ggplot, but it seems the majority of solutions have been implemented in R with ggplot, with a handful in Python and one in d3. My attempts so far have all used R. Unlike this question, I'm interested in displaying specific ranges of data without quantizing/binning the data. That is, I'd like to display a spiral timeline showing when particular events start and stop, where theta-min and theta-max of every event represent specific points in time.
Consider this travel data:
trip_start trip_stop dist
2017-04-01 17:42:00 2017-04-01 18:34:00 1.95
2017-04-01 18:42:00 2017-04-01 19:05:00 6.54
2017-04-02 01:09:00 2017-04-02 01:12:00 1.07
2017-04-02 01:22:00 2017-04-02 01:27:00 1.03
2017-04-02 08:17:00 2017-04-02 08:23:00 1.98
2017-04-02 11:23:00 2017-04-02 11:30:00 1.98
2017-04-02 15:44:00 2017-04-02 15:56:00 4.15
2017-04-02 16:29:00 2017-04-02 16:45:00 4.08
2017-04-03 10:24:00 2017-04-03 10:55:00 19.76
2017-04-03 14:01:00 2017-04-03 14:18:00 8.21
2017-04-03 14:25:00 2017-04-03 14:31:00 1.49
2017-04-03 14:45:00 2017-04-03 14:50:00 1.59
2017-04-03 15:44:00 2017-04-03 16:10:00 4.44
2017-04-03 16:14:00 2017-04-03 16:37:00 9.96
2017-04-03 16:40:00 2017-04-03 16:45:00 0.7
2017-04-03 17:15:00 2017-04-03 17:46:00 16.92
2017-04-03 17:56:00 2017-04-03 18:19:00 5.23
2017-04-03 18:42:00 2017-04-03 18:45:00 0.49
2017-04-03 19:02:00 2017-04-03 19:04:00 0.48
2017-04-04 07:24:00 2017-04-04 07:27:00 0.66
2017-04-04 07:30:00 2017-04-04 08:04:00 13.55
2017-04-04 08:32:00 2017-04-04 09:25:00 25.09
2017-04-04 13:32:00 2017-04-04 13:40:00 3.06
2017-04-04 13:52:00 2017-04-04 13:57:00 1.3
2017-04-04 14:55:00 2017-04-04 15:01:00 2.47
2017-04-04 18:40:00 2017-04-04 19:12:00 22.71
2017-04-04 22:16:00 2017-04-04 23:54:00 38.28
2017-04-04 23:59:00 2017-04-05 00:03:00 1.02
2017-04-05 11:04:00 2017-04-05 11:49:00 25.73
2017-04-05 12:05:00 2017-04-05 12:18:00 2.97
2017-04-05 15:19:00 2017-04-05 16:25:00 25.13
2017-04-05 16:38:00 2017-04-05 16:40:00 0.41
2017-04-05 18:58:00 2017-04-05 19:02:00 1.25
2017-04-05 19:13:00 2017-04-05 19:18:00 1.09
2017-04-05 19:25:00 2017-04-05 19:48:00 6.63
2017-04-06 10:01:00 2017-04-06 10:44:00 20.81
2017-04-06 13:22:00 2017-04-06 13:33:00 1.63
2017-04-06 20:58:00 2017-04-06 21:25:00 24.85
2017-04-06 21:32:00 2017-04-06 21:56:00 6.06
2017-04-07 10:55:00 2017-04-07 11:37:00 24.53
2017-04-07 17:14:00 2017-04-07 17:48:00 19.66
2017-04-07 17:57:00 2017-04-07 18:07:00 2.12
2017-04-08 20:57:00 2017-04-08 21:06:00 1.06
2017-04-08 21:23:00 2017-04-08 21:36:00 2.97
2017-04-09 08:14:00 2017-04-09 08:19:00 1.99
2017-04-09 11:40:00 2017-04-09 11:50:00 2.24
2017-04-09 11:50:00 2017-04-09 11:57:00 1.64
2017-04-09 16:29:00 2017-04-09 16:34:00 0.53
2017-04-09 16:43:00 2017-04-09 16:45:00 0.5
2017-04-09 17:46:00 2017-04-09 17:48:00 0.44
2017-04-09 17:53:00 2017-04-09 17:56:00 0.4
2017-04-09 21:33:00 2017-04-09 21:56:00 2.48
2017-04-09 21:57:00 2017-04-09 22:14:00 2.92
2017-04-09 22:22:00 2017-04-09 22:25:00 0.9
2017-04-10 10:37:00 2017-04-10 11:22:00 19.27
2017-04-10 16:12:00 2017-04-10 16:59:00 21.31
2017-04-11 11:14:00 2017-04-11 11:18:00 1.24
2017-04-11 11:21:00 2017-04-11 11:48:00 22.95
2017-04-11 18:24:00 2017-04-11 19:05:00 28.64
2017-04-11 19:21:00 2017-04-11 19:34:00 5.37
2017-04-12 11:00:00 2017-04-12 12:08:00 28.91
2017-04-12 14:03:00 2017-04-12 15:20:00 28.56
2017-04-12 20:24:00 2017-04-12 20:29:00 1.17
2017-04-12 20:32:00 2017-04-12 21:09:00 30.89
2017-04-13 01:37:00 2017-04-13 02:09:00 32.3
2017-04-13 08:08:00 2017-04-13 08:39:00 19.39
2017-04-13 10:53:00 2017-04-13 11:23:00 24.59
2017-04-13 18:56:00 2017-04-13 19:22:00 22.74
2017-04-14 01:06:00 2017-04-14 01:37:00 31.36
2017-04-14 01:48:00 2017-04-14 01:51:00 1.03
2017-04-14 12:08:00 2017-04-14 12:22:00 1.94
2017-04-14 12:29:00 2017-04-14 13:01:00 19.07
2017-04-14 16:17:00 2017-04-14 17:03:00 19.74
2017-04-14 17:05:00 2017-04-14 17:32:00 3.99
2017-04-14 21:57:00 2017-04-14 22:02:00 1.98
2017-04-15 01:46:00 2017-04-15 01:49:00 1.07
2017-04-15 01:56:00 2017-04-15 01:58:00 1.03
2017-04-15 07:13:00 2017-04-15 07:15:00 0.45
2017-04-15 07:19:00 2017-04-15 07:21:00 0.41
2017-04-15 15:54:00 2017-04-15 16:05:00 1.94
2017-04-15 22:23:00 2017-04-15 22:26:00 0.86
2017-04-15 22:46:00 2017-04-15 22:47:00 0.25
2017-04-15 22:51:00 2017-04-15 22:53:00 0.71
2017-04-16 11:35:00 2017-04-16 11:54:00 11.4
2017-04-16 11:58:00 2017-04-16 12:15:00 10.43
2017-04-17 10:44:00 2017-04-17 10:53:00 3.04
2017-04-17 10:55:00 2017-04-17 11:22:00 18.26
2017-04-17 18:09:00 2017-04-17 18:12:00 0.85
2017-04-17 18:21:00 2017-04-17 19:07:00 37.22
2017-04-18 02:07:00 2017-04-18 02:47:00 32.41
2017-04-18 10:55:00 2017-04-18 10:57:00 0.41
2017-04-18 11:02:00 2017-04-18 11:12:00 2.3
2017-04-18 11:15:00 2017-04-18 11:52:00 24.05
2017-04-18 16:59:00 2017-04-18 17:55:00 22.66
2017-04-19 00:46:00 2017-04-19 01:35:00 39.25
2017-04-19 10:57:00 2017-04-19 11:44:00 24.06
2017-04-19 13:23:00 2017-04-19 14:10:00 25.96
2017-04-19 16:21:00 2017-04-19 17:07:00 18.05
2017-04-19 23:32:00 2017-04-20 00:19:00 39.67
2017-04-20 10:47:00 2017-04-20 11:13:00 24.07
2017-04-20 16:21:00 2017-04-20 16:30:00 0.86
2017-04-20 16:36:00 2017-04-20 16:58:00 0.85
2017-04-20 17:41:00 2017-04-20 17:44:00 0.37
2017-04-20 17:49:00 2017-04-20 18:40:00 19.32
2017-04-20 22:22:00 2017-04-20 22:53:00 29.2
2017-04-20 23:07:00 2017-04-20 23:27:00 10.94
2017-04-21 08:29:00 2017-04-21 08:40:00 1.91
2017-04-21 11:30:00 2017-04-21 11:32:00 0.42
2017-04-21 11:38:00 2017-04-21 11:40:00 0.4
2017-04-21 11:42:00 2017-04-21 12:15:00 19.09
2017-04-21 16:50:00 2017-04-21 18:17:00 40.61
2017-04-21 18:55:00 2017-04-21 19:11:00 1.73
2017-04-21 22:20:00 2017-04-21 22:53:00 28.26
2017-04-21 23:01:00 2017-04-21 23:22:00 11.76
2017-04-22 08:56:00 2017-04-22 08:58:00 0.63
2017-04-22 09:04:00 2017-04-22 09:08:00 0.3
2017-04-22 09:12:00 2017-04-22 09:15:00 0.42
2017-04-22 16:48:00 2017-04-22 16:52:00 0.54
2017-04-22 17:06:00 2017-04-22 17:09:00 0.51
2017-04-22 17:10:00 2017-04-22 17:13:00 1.03
2017-04-22 17:22:00 2017-04-22 17:27:00 1.1
2017-04-23 08:13:00 2017-04-23 08:15:00 0.41
2017-04-23 08:19:00 2017-04-23 08:20:00 0.4
2017-04-23 08:21:00 2017-04-23 08:25:00 1.99
2017-04-23 11:41:00 2017-04-23 11:48:00 2.04
2017-04-23 12:35:00 2017-04-23 12:50:00 7.59
2017-04-23 14:08:00 2017-04-23 14:21:00 7.31
2017-04-23 14:33:00 2017-04-23 15:38:00 37.6
2017-04-24 00:26:00 2017-04-24 01:18:00 39.21
2017-04-24 10:24:00 2017-04-24 10:26:00 0.41
2017-04-24 10:31:00 2017-04-24 10:35:00 1.37
2017-04-24 10:38:00 2017-04-24 10:43:00 1.19
2017-04-24 10:49:00 2017-04-24 11:15:00 19.58
2017-04-24 17:13:00 2017-04-24 18:20:00 37.42
2017-04-24 19:02:00 2017-04-24 19:08:00 1.76
2017-04-24 19:49:00 2017-04-24 19:55:00 1.79
2017-04-24 20:41:00 2017-04-24 21:16:00 32.31
2017-04-25 10:53:00 2017-04-25 11:25:00 24.83
2017-04-25 15:15:00 2017-04-25 15:24:00 3.07
2017-04-25 15:30:00 2017-04-25 15:40:00 3.01
2017-04-25 17:34:00 2017-04-25 18:18:00 24.8
2017-04-26 09:59:00 2017-04-26 10:28:00 24.05
2017-04-26 12:56:00 2017-04-26 13:40:00 29.13
2017-04-26 14:37:00 2017-04-26 15:34:00 21
2017-04-27 08:57:00 2017-04-27 10:21:00 40.56
2017-04-27 16:12:00 2017-04-27 16:44:00 9.89
2017-04-27 17:09:00 2017-04-27 18:01:00 17.51
2017-04-28 05:18:00 2017-04-28 06:06:00 39.28
2017-04-28 12:57:00 2017-04-28 13:52:00 35.82
2017-04-28 16:48:00 2017-04-28 18:14:00 39.1
2017-05-01 11:41:00 2017-05-01 12:20:00 18.74
2017-05-01 18:53:00 2017-05-01 19:34:00 37.15
2017-05-01 23:08:00 2017-05-01 23:09:00 0.06
2017-05-01 23:18:00 2017-05-02 00:11:00 38.61
2017-05-02 11:05:00 2017-05-02 11:42:00 24.07
2017-05-02 17:34:00 2017-05-02 18:53:00 26.42
2017-05-03 12:13:00 2017-05-03 12:25:00 3.96
2017-05-03 12:25:00 2017-05-03 12:56:00 21.15
2017-05-03 13:26:00 2017-05-03 13:44:00 3.32
2017-05-03 13:57:00 2017-05-03 14:08:00 3.49
2017-05-03 18:39:00 2017-05-03 19:08:00 24.85
2017-05-03 19:09:00 2017-05-03 19:13:00 0.99
2017-05-03 19:29:00 2017-05-03 19:32:00 0.84
2017-05-04 10:38:00 2017-05-04 11:06:00 24.05
2017-05-04 13:34:00 2017-05-04 14:10:00 1.73
2017-05-04 17:14:00 2017-05-04 18:23:00 24.68
2017-05-05 20:38:00 2017-05-05 20:52:00 2.24
2017-05-06 11:45:00 2017-05-06 12:30:00 20.19
2017-05-06 14:36:00 2017-05-06 15:35:00 14.49
2017-05-06 15:48:00 2017-05-06 16:17:00 5.25
2017-05-06 17:11:00 2017-05-06 17:13:00 0.43
2017-05-06 17:19:00 2017-05-06 17:21:00 0.43
2017-05-07 08:16:00 2017-05-07 08:22:00 3.27
2017-05-07 12:09:00 2017-05-07 12:16:00 2.01
2017-05-07 17:28:00 2017-05-07 17:50:00 10.36
2017-05-07 17:54:00 2017-05-07 18:01:00 1.19
2017-05-07 18:02:00 2017-05-07 18:35:00 28.31
2017-05-07 21:48:00 2017-05-07 21:52:00 1.46
2017-05-07 22:01:00 2017-05-07 22:05:00 1.37
2017-05-08 00:59:00 2017-05-08 02:19:00 39.23
2017-05-08 11:30:00 2017-05-08 11:58:00 22.55
2017-05-08 18:08:00 2017-05-08 18:30:00 10.47
2017-05-08 18:33:00 2017-05-08 19:09:00 28.44
2017-05-08 22:25:00 2017-05-08 23:09:00 38.65
2017-05-08 23:14:00 2017-05-08 23:17:00 1.04
2017-05-09 11:35:00 2017-05-09 12:19:00 23.99
2017-05-09 17:57:00 2017-05-09 18:59:00 29.38
2017-05-09 20:03:00 2017-05-09 20:13:00 1.9
2017-05-10 10:18:00 2017-05-10 10:54:00 24.06
2017-05-10 15:43:00 2017-05-10 16:46:00 24.71
2017-05-11 12:28:00 2017-05-11 13:07:00 21.75
2017-05-11 18:00:00 2017-05-11 18:31:00 19.3
2017-05-12 08:26:00 2017-05-12 08:55:00 20.46
2017-05-12 13:00:00 2017-05-12 13:34:00 14.6
2017-05-13 08:44:00 2017-05-13 08:46:00 0.38
2017-05-13 08:57:00 2017-05-13 09:01:00 0.33
2017-05-13 14:22:00 2017-05-13 14:41:00 6.86
2017-05-13 15:17:00 2017-05-13 15:35:00 5.2
2017-05-13 18:10:00 2017-05-13 18:21:00 1.91
2017-05-14 11:22:00 2017-05-14 11:26:00 0.9
2017-05-14 11:36:00 2017-05-14 11:38:00 0.39
2017-05-14 14:56:00 2017-05-14 15:59:00 40.07
2017-05-14 16:34:00 2017-05-14 16:41:00 1.49
2017-05-14 16:56:00 2017-05-14 17:04:00 1.45
2017-05-14 19:05:00 2017-05-14 20:06:00 39.21
2017-05-15 11:24:00 2017-05-15 11:33:00 1.91
2017-05-15 11:41:00 2017-05-15 12:13:00 19.84
2017-05-15 17:41:00 2017-05-15 18:11:00 16
2017-05-15 18:15:00 2017-05-15 19:23:00 31.52
2017-05-15 23:41:00 2017-05-16 00:26:00 39.32
2017-05-16 09:49:00 2017-05-16 11:02:00 24.91
2017-05-16 16:08:00 2017-05-16 16:32:00 3.37
2017-05-16 17:11:00 2017-05-16 17:32:00 4.8
2017-05-16 17:42:00 2017-05-16 17:56:00 1.81
2017-05-16 18:13:00 2017-05-16 18:46:00 24.85
2017-05-16 21:07:00 2017-05-16 21:10:00 1.04
2017-05-16 21:26:00 2017-05-16 21:29:00 1.02
2017-07-28 16:10:00 2017-07-28 16:17:00 2.22
2017-07-28 16:17:00 2017-07-28 16:42:00 7.84
2017-08-10 12:00:00 2017-08-10 12:44:00 24.05
2017-08-10 14:56:00 2017-08-10 15:10:00 1.61
2017-08-10 18:51:00 2017-08-10 19:21:00 24.85
2017-08-10 19:46:00 2017-08-10 19:56:00 1.14
2017-08-10 20:08:00 2017-08-10 20:12:00 1.09
2017-08-11 12:44:00 2017-08-11 12:49:00 0.82
2017-08-11 12:59:00 2017-08-11 13:01:00 0.56
2017-08-11 13:18:00 2017-08-11 15:12:00 1.79
2017-08-11 15:14:00 2017-08-11 16:53:00 34.6
2017-08-11 19:27:00 2017-08-11 20:34:00 34.91
2017-08-12 13:52:00 2017-08-12 13:56:00 1.05
2017-08-12 13:59:00 2017-08-12 14:02:00 0.28
2017-08-12 14:10:00 2017-08-12 14:30:00 1.22
2017-08-12 17:15:00 2017-08-12 17:36:00 11.37
2017-08-12 20:49:00 2017-08-12 21:05:00 10.43
2017-08-13 12:16:00 2017-08-13 12:44:00 12.96
2017-08-13 16:03:00 2017-08-13 16:32:00 14.33
2017-08-13 18:19:00 2017-08-13 18:42:00 9.32
2017-08-13 18:52:00 2017-08-13 19:05:00 3.99
2017-08-13 21:42:00 2017-08-13 21:53:00 5.6
2017-08-14 08:50:00 2017-08-14 09:45:00 24.1
2017-08-14 13:22:00 2017-08-14 13:54:00 24.84
2017-08-14 14:02:00 2017-08-14 15:34:00 36.92
2017-08-14 15:58:00 2017-08-14 17:17:00 35.7
2017-08-14 17:35:00 2017-08-14 17:45:00 1.99
2017-08-14 18:07:00 2017-08-14 18:27:00 9.92
2017-08-15 10:15:00 2017-08-15 10:51:00 25
2017-08-15 19:23:00 2017-08-15 19:29:00 0.4
2017-08-15 19:51:00 2017-08-15 20:45:00 24.39
2017-08-15 20:56:00 2017-08-15 21:04:00 2.78
2017-08-15 21:09:00 2017-08-15 21:37:00 19.22
2017-08-16 00:03:00 2017-08-16 00:27:00 15.51
2017-08-16 00:36:00 2017-08-16 00:41:00 1.23
2017-08-16 00:46:00 2017-08-16 01:18:00 11.35
2017-08-16 09:38:00 2017-08-16 09:41:00 1.21
2017-08-16 09:41:00 2017-08-16 09:43:00 0.08
2017-08-16 09:47:00 2017-08-16 10:32:00 22.89
2017-08-16 16:51:00 2017-08-16 17:11:00 3.14
2017-08-16 17:12:00 2017-08-16 17:25:00 2.76
2017-08-16 17:41:00 2017-08-16 18:36:00 24.78
2017-08-17 09:34:00 2017-08-17 10:13:00 24.03
2017-08-17 12:32:00 2017-08-17 13:07:00 24.82
2017-08-17 13:35:00 2017-08-17 13:40:00 0.4
2017-08-17 13:47:00 2017-08-17 15:07:00 36.06
2017-08-17 15:18:00 2017-08-17 15:24:00 0.06
2017-08-17 16:03:00 2017-08-17 18:05:00 35.16
2017-08-18 09:47:00 2017-08-18 10:23:00 24.47
2017-08-18 16:04:00 2017-08-18 16:42:00 1.63
2017-08-18 17:56:00 2017-08-18 18:25:00 10.74
2017-08-18 18:27:00 2017-08-18 18:48:00 1.85
2017-08-19 00:07:00 2017-08-19 00:41:00 18.92
2017-08-19 00:52:00 2017-08-19 00:55:00 0.99
2017-08-19 11:52:00 2017-08-19 12:14:00 7.56
2017-08-19 15:57:00 2017-08-19 16:12:00 4.02
2017-08-19 16:37:00 2017-08-19 16:56:00 5.32
2017-08-19 23:32:00 2017-08-19 23:50:00 7.54
2017-08-19 23:51:00 2017-08-20 00:17:00 9.59
2017-08-20 09:03:00 2017-08-20 09:16:00 5.22
2017-08-20 19:17:00 2017-08-20 19:32:00 4.69
2017-08-21 09:24:00 2017-08-21 09:40:00 2.31
2017-08-21 10:59:00 2017-08-21 11:02:00 0.47
2017-08-21 13:40:00 2017-08-21 15:29:00 36.09
2017-08-21 15:54:00 2017-08-21 16:48:00 2.24
2017-08-21 16:57:00 2017-08-21 18:15:00 32.3
2017-08-22 08:38:00 2017-08-22 09:06:00 0.65
2017-08-22 09:18:00 2017-08-22 09:19:00 0.04
2017-08-22 09:22:00 2017-08-22 10:05:00 23.49
2017-08-22 14:30:00 2017-08-22 15:02:00 1.7
2017-08-22 16:37:00 2017-08-22 17:41:00 24.8
2017-08-23 17:16:00 2017-08-23 18:14:00 24.01
2017-08-23 18:27:00 2017-08-23 18:32:00 1.05
2017-08-23 19:24:00 2017-08-23 20:04:00 18.14
2017-08-23 22:01:00 2017-08-23 22:28:00 16.33
2017-08-23 22:46:00 2017-08-23 22:50:00 1.04
2017-08-24 09:41:00 2017-08-24 09:44:00 0.02
2017-08-24 09:59:00 2017-08-24 10:00:00 0.02
2017-08-24 13:57:00 2017-08-24 15:33:00 42.51
2017-08-24 16:43:00 2017-08-24 17:00:00 0.07
2017-08-24 17:06:00 2017-08-24 17:33:00 10.01
2017-08-24 18:12:00 2017-08-24 19:03:00 27.67
2017-08-25 09:36:00 2017-08-25 09:55:00 2.63
2017-08-25 10:01:00 2017-08-25 10:32:00 20.92
2017-08-25 20:40:00 2017-08-25 21:45:00 17.41
2017-08-25 21:49:00 2017-08-25 22:14:00 16.02
2017-08-26 00:10:00 2017-08-26 02:14:00 29.77
2017-08-26 16:31:00 2017-08-26 16:55:00 7.15
2017-08-26 17:54:00 2017-08-26 18:19:00 10
2017-08-26 20:07:00 2017-08-26 20:08:00 0.19
2017-08-26 20:08:00 2017-08-26 20:11:00 1.35
2017-08-27 12:39:00 2017-08-27 12:54:00 1
2017-08-27 12:55:00 2017-08-27 13:48:00 9.29
2017-08-27 14:00:00 2017-08-27 14:34:00 3.86
2017-08-27 15:56:00 2017-08-27 16:37:00 10.45
2017-08-27 16:44:00 2017-08-27 16:51:00 1.8
2017-08-27 16:55:00 2017-08-27 17:00:00 0.68
2017-08-27 17:04:00 2017-08-27 17:19:00 4.96
2017-08-27 17:28:00 2017-08-27 17:39:00 2.33
2017-08-27 17:47:00 2017-08-27 18:58:00 24.19
2017-08-27 22:17:00 2017-08-27 22:41:00 16.24
2017-08-28 00:33:00 2017-08-28 01:22:00 13.62
2017-08-28 12:48:00 2017-08-28 12:51:00 0.47
2017-08-28 14:01:00 2017-08-28 14:03:00 0.4
2017-08-28 14:12:00 2017-08-28 15:31:00 34.86
2017-08-28 15:56:00 2017-08-28 17:04:00 34.47
2017-08-28 22:15:00 2017-08-28 22:38:00 18.57
2017-08-29 01:42:00 2017-08-29 02:05:00 18.88
2017-08-29 11:40:00 2017-08-29 11:44:00 1.04
2017-08-29 11:48:00 2017-08-29 12:09:00 0.03
2017-08-29 12:18:00 2017-08-29 12:21:00 0.03
2017-08-29 12:26:00 2017-08-29 12:32:00 1.05
2017-08-29 12:35:00 2017-08-29 13:15:00 24.05
2017-08-29 19:40:00 2017-08-29 19:42:00 0.35
2017-08-29 19:50:00 2017-08-29 20:19:00 27.72
2017-08-29 20:25:00 2017-08-29 20:41:00 10.42
2017-08-30 10:00:00 2017-08-30 10:47:00 24.25
2017-08-30 14:31:00 2017-08-30 14:56:00 1.68
2017-08-30 17:19:00 2017-08-30 17:43:00 0.04
2017-08-30 17:43:00 2017-08-30 17:50:00 0.29
2017-08-30 17:56:00 2017-08-30 18:40:00 16.85
2017-08-30 22:57:00 2017-08-30 23:35:00 17.31
2017-08-31 11:30:00 2017-08-31 11:41:00 0.43
2017-08-31 14:04:00 2017-08-31 14:06:00 0.41
2017-08-31 14:24:00 2017-08-31 14:26:00 0.68
2017-08-31 14:31:00 2017-08-31 15:42:00 34.88
2017-08-31 16:01:00 2017-08-31 17:07:00 30.45
2017-08-31 20:54:00 2017-08-31 21:21:00 19.6
2017-09-01 10:30:00 2017-09-01 10:59:00 17.63
2017-09-01 14:07:00 2017-09-01 15:07:00 27.45
2017-09-01 17:17:00 2017-09-01 17:36:00 1.93
2017-09-01 18:16:00 2017-09-01 19:19:00 20.58
2017-09-01 19:25:00 2017-09-01 19:38:00 4.8
2017-09-01 21:30:00 2017-09-01 21:54:00 1.94
2017-09-02 15:46:00 2017-09-02 16:06:00 0.99
2017-09-02 16:13:00 2017-09-02 16:16:00 1.01
2017-09-02 16:56:00 2017-09-02 16:59:00 0.42
2017-09-02 17:04:00 2017-09-02 17:06:00 0.4
2017-09-02 22:52:00 2017-09-02 22:54:00 0.07
2017-09-02 22:55:00 2017-09-02 23:15:00 18.62
2017-09-03 01:46:00 2017-09-03 02:10:00 18.9
2017-09-03 14:49:00 2017-09-03 15:04:00 3.14
2017-09-03 15:50:00 2017-09-03 16:07:00 10.17
2017-09-03 16:21:00 2017-09-03 16:38:00 7.79
2017-09-03 16:47:00 2017-09-03 16:52:00 1.11
2017-09-03 18:32:00 2017-09-03 18:37:00 1.2
2017-09-03 18:37:00 2017-09-03 18:44:00 0.91
2017-09-04 15:50:00 2017-09-04 15:54:00 0.42
2017-09-04 15:59:00 2017-09-04 16:11:00 2.3
2017-09-04 16:21:00 2017-09-04 16:43:00 8.31
2017-09-04 17:05:00 2017-09-04 17:15:00 2.54
2017-09-04 17:26:00 2017-09-04 17:41:00 4.52
2017-09-04 17:49:00 2017-09-04 18:25:00 29.55
2017-09-04 19:36:00 2017-09-04 19:51:00 0.93
2017-09-04 19:54:00 2017-09-04 19:59:00 0.5
2017-09-04 21:21:00 2017-09-04 21:55:00 29.37
2017-09-05 11:08:00 2017-09-05 11:51:00 35.5
2017-09-05 12:36:00 2017-09-05 13:07:00 2.29
2017-09-05 13:19:00 2017-09-05 13:22:00 0.51
2017-09-05 13:26:00 2017-09-05 14:03:00 33.09
2017-09-05 14:13:00 2017-09-05 15:01:00 24.03
2017-09-05 17:33:00 2017-09-05 18:11:00 14.55
2017-09-05 19:01:00 2017-09-05 19:19:00 11.31
2017-09-06 09:21:00 2017-09-06 09:39:00 7.73
2017-09-06 10:14:00 2017-09-06 10:30:00 7.75
2017-09-06 10:37:00 2017-09-06 11:13:00 24.13
2017-09-06 16:48:00 2017-09-06 17:35:00 25.3
2017-09-06 17:49:00 2017-09-06 17:55:00 0.18
2017-09-06 17:58:00 2017-09-06 18:00:00 0.39
2017-09-06 18:38:00 2017-09-06 19:04:00 15.93
2017-09-06 23:45:00 2017-09-07 00:14:00 19.45
2017-09-07 00:26:00 2017-09-07 00:30:00 1.01
2017-09-07 10:42:00 2017-09-07 11:35:00 31.74
2017-09-07 14:04:00 2017-09-07 14:39:00 27.38
2017-09-07 14:43:00 2017-09-07 14:52:00 3.06
2017-09-07 14:54:00 2017-09-07 16:00:00 32.96
2017-09-07 16:32:00 2017-09-07 16:33:00 0.07
2017-09-07 16:38:00 2017-09-07 17:04:00 2.31
2017-09-07 17:23:00 2017-09-07 18:14:00 33.03
2017-09-08 10:02:00 2017-09-08 10:30:00 19.73
2017-09-08 18:09:00 2017-09-08 18:37:00 18.97
2017-09-08 19:04:00 2017-09-08 19:18:00 1.87
2017-09-09 02:25:00 2017-09-09 02:28:00 1.1
2017-09-09 02:33:00 2017-09-09 02:35:00 1.05
2017-09-10 17:09:00 2017-09-10 17:44:00 14.25
2017-09-10 22:50:00 2017-09-10 22:53:00 0.25
2017-09-10 22:56:00 2017-09-10 22:57:00 0.02
2017-09-10 23:00:00 2017-09-10 23:23:00 16.18
2017-09-11 00:01:00 2017-09-11 00:19:00 1.83
2017-09-11 09:59:00 2017-09-11 10:06:00 1.91
2017-09-11 10:12:00 2017-09-11 10:51:00 27.49
2017-09-11 13:39:00 2017-09-11 14:13:00 27.23
2017-09-11 14:31:00 2017-09-11 15:31:00 35.45
2017-09-11 16:03:00 2017-09-11 17:09:00 36.01
2017-09-11 17:39:00 2017-09-11 18:01:00 9.88
2017-09-11 23:01:00 2017-09-11 23:05:00 1.14
2017-09-11 23:16:00 2017-09-11 23:30:00 5.93
2017-09-11 23:30:00 2017-09-11 23:54:00 4.94
2017-09-12 02:56:00 2017-09-12 04:00:00 25.87
2017-09-12 10:06:00 2017-09-12 10:46:00 24.84
2017-09-12 16:33:00 2017-09-12 17:20:00 22.43
2017-09-12 19:38:00 2017-09-12 20:14:00 21.79
2017-09-13 06:24:00 2017-09-13 06:59:00 25.84
2017-09-13 07:02:00 2017-09-13 07:14:00 5.77
2017-09-13 11:14:00 2017-09-13 11:36:00 16.26
2017-09-13 16:01:00 2017-09-13 16:57:00 24.79
2017-09-13 17:07:00 2017-09-13 17:48:00 15.94
2017-09-13 23:13:00 2017-09-13 23:35:00 16.73
2017-09-14 12:00:00 2017-09-14 12:27:00 19.71
2017-09-14 12:28:00 2017-09-14 12:30:00 0.18
2017-09-14 14:36:00 2017-09-14 15:06:00 14.98
2017-09-14 15:11:00 2017-09-14 15:17:00 2.99
2017-09-14 15:26:00 2017-09-14 16:44:00 37.48
2017-09-14 17:03:00 2017-09-14 18:17:00 34.18
2017-09-14 18:32:00 2017-09-14 18:41:00 3.03
2017-09-15 10:25:00 2017-09-15 10:26:00 0.05
2017-09-15 10:45:00 2017-09-15 10:48:00 0.29
2017-09-15 10:59:00 2017-09-15 11:05:00 0.3
2017-09-15 11:09:00 2017-09-15 11:36:00 10.82
2017-09-15 13:00:00 2017-09-15 13:17:00 8.37
2017-09-15 13:36:00 2017-09-15 14:30:00 25.19
2017-09-15 14:37:00 2017-09-15 15:01:00 0.45
2017-09-15 15:04:00 2017-09-15 16:59:00 85.51
2017-09-15 17:06:00 2017-09-15 18:57:00 129.72
2017-09-15 19:03:00 2017-09-15 20:02:00 60.96
2017-09-16 10:18:00 2017-09-16 10:39:00 16.04
2017-09-16 11:52:00 2017-09-16 12:12:00 16.68
2017-09-16 12:28:00 2017-09-16 13:29:00 49
2017-09-16 18:36:00 2017-09-16 19:30:00 45.7
2017-09-16 19:39:00 2017-09-16 19:47:00 2.1
2017-09-17 13:32:00 2017-09-17 13:41:00 2.24
2017-09-17 14:19:00 2017-09-17 14:48:00 14.68
2017-09-17 18:25:00 2017-09-17 18:26:00 0.05
2017-09-17 18:36:00 2017-09-17 19:03:00 12.26
2017-09-18 07:52:00 2017-09-18 08:03:00 2.04
2017-09-18 08:21:00 2017-09-18 08:56:00 37.94
2017-09-18 09:01:00 2017-09-18 09:53:00 65.7
2017-09-18 10:04:00 2017-09-18 10:34:00 39.43
2017-09-18 10:46:00 2017-09-18 11:07:00 14.25
2017-09-18 11:19:00 2017-09-18 13:29:00 138.98
2017-09-18 14:24:00 2017-09-18 14:26:00 0.04
2017-09-18 14:28:00 2017-09-18 15:23:00 35.52
2017-09-18 15:53:00 2017-09-18 17:49:00 36.64
2017-09-19 09:24:00 2017-09-19 10:22:00 24.37
2017-09-19 15:55:00 2017-09-19 16:53:00 15.87
2017-09-19 16:53:00 2017-09-19 17:20:00 0.85
2017-09-19 17:33:00 2017-09-19 18:06:00 10.95
2017-09-19 18:10:00 2017-09-19 18:34:00 8.41
2017-09-19 21:06:00 2017-09-19 21:10:00 1.24
2017-09-19 21:17:00 2017-09-19 21:21:00 1.05
2017-09-20 11:12:00 2017-09-20 11:16:00 1.22
2017-09-20 11:18:00 2017-09-20 11:59:00 24.15
2017-09-20 17:20:00 2017-09-20 18:07:00 24.15
2017-09-20 18:50:00 2017-09-20 19:17:00 16.02
2017-09-20 22:05:00 2017-09-20 22:32:00 17.5
2017-09-21 13:38:00 2017-09-21 13:44:00 0.72
2017-09-21 13:50:00 2017-09-21 15:26:00 35.81
2017-09-21 15:59:00 2017-09-21 16:15:00 8.26
2017-09-21 16:19:00 2017-09-21 17:32:00 28.1
2017-09-21 18:49:00 2017-09-21 19:25:00 16.05
2017-09-21 22:30:00 2017-09-21 22:59:00 16.97
2017-09-22 10:19:00 2017-09-22 10:21:00 0.43
2017-09-22 10:25:00 2017-09-22 10:26:00 0.4
2017-09-22 10:30:00 2017-09-22 10:54:00 19.15
2017-09-22 11:58:00 2017-09-22 12:02:00 1.05
2017-09-22 18:32:00 2017-09-22 18:59:00 20.95
2017-09-23 08:34:00 2017-09-23 08:51:00 1.15
2017-09-23 09:19:00 2017-09-23 10:31:00 37.57
2017-09-23 11:09:00 2017-09-23 11:23:00 5.67
2017-09-23 11:51:00 2017-09-23 12:15:00 4.64
2017-09-23 12:47:00 2017-09-23 13:40:00 8.45
2017-09-23 13:56:00 2017-09-23 15:08:00 34.62
2017-09-23 15:37:00 2017-09-23 16:07:00 1.56
2017-09-24 14:59:00 2017-09-24 15:02:00 0.43
2017-09-24 15:14:00 2017-09-24 17:09:00 6.6
2017-09-24 17:37:00 2017-09-24 18:01:00 7.05
2017-09-24 18:05:00 2017-09-24 18:07:00 0.41
2017-09-24 19:35:00 2017-09-24 20:31:00 25.28
2017-09-25 00:24:00 2017-09-25 00:26:00 0.42
2017-09-25 00:30:00 2017-09-25 01:10:00 23.13
2017-09-25 12:12:00 2017-09-25 12:38:00 19.45
2017-09-25 14:22:00 2017-09-25 14:50:00 19.86
2017-09-25 14:52:00 2017-09-25 15:54:00 35.53
2017-09-25 16:37:00 2017-09-25 18:17:00 34.54
2017-09-25 20:36:00 2017-09-25 21:08:00 28.91
2017-09-26 01:46:00 2017-09-26 02:21:00 26.32
2017-09-26 09:36:00 2017-09-26 10:18:00 24.02
2017-09-26 14:05:00 2017-09-26 14:39:00 25.3
2017-09-26 15:49:00 2017-09-26 15:58:00 1.53
2017-09-26 16:15:00 2017-09-26 16:22:00 1.1
2017-09-27 09:15:00 2017-09-27 10:16:00 24.76
2017-09-27 16:26:00 2017-09-27 17:49:00 35.87
2017-09-27 17:58:00 2017-09-27 18:46:00 27.64
2017-09-27 18:51:00 2017-09-27 18:59:00 2.08
2017-09-27 19:10:00 2017-09-27 20:17:00 21.17
2017-09-27 20:25:00 2017-09-27 21:56:00 3.6
2017-09-27 22:04:00 2017-09-27 22:32:00 16.56
2017-09-28 06:46:00 2017-09-28 07:19:00 14.4
2017-09-28 09:05:00 2017-09-28 09:29:00 8.06
2017-09-28 10:41:00 2017-09-28 11:21:00 22.34
2017-09-28 14:26:00 2017-09-28 16:05:00 35.57
2017-09-28 16:09:00 2017-09-28 16:21:00 1.17
2017-09-28 20:37:00 2017-09-28 20:40:00 1.1
2017-09-28 20:56:00 2017-09-28 21:00:00 1.15
2017-09-29 09:32:00 2017-09-29 10:02:00 19.73
I'd like to plot these discrete events the same way the below plots do, but where 2pi is one week rather than 24 hours in order to illuminate the periodicity of these events, where color represents distance.
I've attempted modifying the solution linked at the beginning of this question, but it hasn't gotten me anywhere. My new approach is to modify this solution, but I'm having a difficult time getting anything but horizontal and vertical lines scattered about a spiral. Making them curve and display in the correct locations is tough.
I'm open to any approach that successfully displays the data in a spiral plot without quantizing/binning it into specific intervals but rather allows the intervals themselves to describe discrete events along a continuous spiralling timeline. Likewise, I'm not interested in converting this to a raw single-point time series format where I'd have a great deal of data representing the time between trips. I'd like to achieve this in a temporal format (one that describes a time window rather than an event at a particular time).
Still needs work, but it's a start, with python and matplotlib.
The idea is to plot a spiral timeline in polar coordinates with 1 week period, each event is an arc of this spiral with a color depending on dist data.
There are lots of overlapping intervals though that this visualization tends to hide... maybe semitransparent arcs could be better, with a carefully chosen colormap.
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.patheffects as mpe
import pandas as pd
# styling
LINEWIDTH=4
EDGEWIDTH=1
CAPSTYLE="projecting"
COLORMAP="viridis_r"
ALPHA=1
FIRSTDAY=6 # 0=Mon, 6=Sun
# load dataset and parse timestamps
df = pd.read_csv('trips.csv')
df[['trip_start', 'trip_stop']] = df[['trip_start', 'trip_stop']].apply(pd.to_datetime)
# set origin at the first FIRSTDAY before the first trip, midnight
first_trip = df['trip_start'].min()
origin = (first_trip - pd.to_timedelta(first_trip.weekday() - FIRSTDAY, unit='d')).replace(hour=0, minute=0, second=0)
weekdays = pd.date_range(origin, origin + np.timedelta64(1, 'W')).strftime("%a").tolist()
# # convert trip timestamps to week fractions
df['start'] = (df['trip_start'] - origin) / np.timedelta64(1, 'W')
df['stop'] = (df['trip_stop'] - origin) / np.timedelta64(1, 'W')
# sort dataset so shortest trips are plotted last
# should prevent longer events to cover shorter ones, still suboptimal
df = df.sort_values('dist', ascending=False).reset_index()
fig = plt.figure(figsize=(8, 6))
ax = fig.gca(projection="polar")
for idx, event in df.iterrows():
# sample normalized distance from colormap
ndist = event['dist'] / df['dist'].max()
color = plt.cm.get_cmap(COLORMAP)(ndist)
tstart, tstop = event.loc[['start', 'stop']]
# timestamps are in week fractions, 2pi is one week
nsamples = int(1000. * (tstop - tstart))
t = np.linspace(tstart, tstop, nsamples)
theta = 2 * np.pi * t
arc, = ax.plot(theta, t, lw=LINEWIDTH, color=color, solid_capstyle=CAPSTYLE, alpha=ALPHA)
if EDGEWIDTH > 0:
arc.set_path_effects([mpe.Stroke(linewidth=LINEWIDTH+EDGEWIDTH, foreground='black'), mpe.Normal()])
# grid and labels
ax.set_rticks([])
ax.set_theta_zero_location("N")
ax.set_theta_direction(-1)
ax.set_xticks(np.linspace(0, 2*np.pi, 7, endpoint=False))
ax.set_xticklabels(weekdays)
ax.tick_params('x', pad=2)
ax.grid(True)
# setup a custom colorbar, everything's always a bit tricky with mpl colorbars
vmin = df['dist'].min()
vmax = df['dist'].max()
norm = mpl.colors.Normalize(vmin=vmin, vmax=vmax)
sm = plt.cm.ScalarMappable(cmap=COLORMAP, norm=norm)
sm.set_array([])
plt.colorbar(sm, ticks=np.linspace(vmin, vmax, 10), fraction=0.04, aspect=60, pad=0.1, label="distance", ax=ax)
plt.savefig("spiral.png", pad_inches=0, bbox_inches="tight")
Full timeline
To see it's a spiral that never overlaps and it works for longer events too you can plot the full timeline (here with LINEWIDTH=3.5 to limit moiré fringing).
fullt = np.linspace(df['start'].min(), df['stop'].max(), 10000)
theta = 2 * np.pi * fullt
ax.plot(theta, fullt, lw=LINEWIDTH,
path_effects=[mpe.Stroke(linewidth=LINEWIDTH+LINEBORDER, foreground='black'), mpe.Normal()])
Example with a random set...
Here's the plot for a random dataset of 200 mainly short trips with the occasional 1 to 2 weeks long ones.
N = 200
df = pd.DataFrame()
df["start"] = np.random.uniform(0, 20, size=N)
df["stop"] = df["start"] + np.random.choice([np.random.uniform(0, 0.1),
np.random.uniform(1., 2.)], p=[0.98, 0.02], size=N)
df["dist"] = np.random.random(size=N)
... and different styles
inferno_r color map, rounded or butted linecaps, semitransparent, bolder edges, etc (click for full size)
Here's a start. Let me know if this is what you had in mind.
I began with your data sample and put trip_start and trip_stop into POSIXct format before continuing with the code below.
library(tidyverse)
library(lubridate)
dat = dat %>%
mutate(start=(hour(trip_start)*60 + minute(trip_start) + second(trip_start))/(24*60) + wday(trip_start),
stop=(hour(trip_stop)*60 + minute(trip_stop) + second(trip_stop))/(24*60) + wday(trip_stop),
tod = case_when(hour(trip_start) < 6 ~ "night",
hour(trip_start) < 12 ~ "morning",
hour(trip_start) < 18 ~ "afternoon",
hour(trip_start) < 24 ~ "evening"))
ggplot(dat) +
geom_segment(aes(x=start, xend=stop,
y=trip_start,
yend=trip_stop,
colour=tod),
size=5, show.legend = FALSE) +
coord_polar() +
scale_y_datetime(breaks=seq(as.POSIXct("2017-09-01"), as.POSIXct("2017-12-31"), by="week")) +
scale_x_continuous(limits=c(1,8), breaks=1:7,
labels=weekdays(x=as.Date(seq(7)+2, origin="1970-01-01"),
abbreviate=TRUE))+
expand_limits(y=as.POSIXct("2017-08-25")) +
theme_bw() +
scale_colour_manual(values=c(night="black", morning="orange",
afternoon="orange", evening="blue")) +
labs(x="",y="")
This could be achieved relatively straightforwardly with d3. I'll use your data to create a rough template of one basic possible approach. Here's what the result of this approach might look like:
The key ingredient is d3's radial line component that lets us define a line by plotting angle and radius (here's a recent answer showing another spiral graph, that answer started me down the path on this answer).
All we need to do is scale angle and radius to be able to use this effectively (for which we need the first time and last time in the dataset):
var angle = d3.scaleTime()
.domain([start,end])
.range([0,Math.PI * 2 * numberWeeks])
var radius = d3.scaleTime()
.domain([start,end])
.range([minInnerRadius,maxOuterRadius])
And from there we can create a spiral quite easily, we sample some dates throughout the interval and then pass them to the radial line function:
var spiral = d3.radialLine()
.curve(d3.curveCardinal)
.angle(angle)
.radius(radius);
Here's a quick demonstration of just the spiral covering your time period. I'm assuming a base familiarity with d3 for this answer, so have not touched on a few parts of the code.
Once we have that, it's just a matter of adding sections from the data. The most simple way would be to plainly draw a stroke with some width and color it appropriately. This requires the same as above, but rather than sampling points from the start and end times of the dataset, we just need the start and end times of each datum:
// append segments on spiral:
var segments = g.selectAll()
.data(data)
.enter()
.append("path")
.attr("d", function(d) {
return /* sample points and feed to spiral function here */;
})
.style("stroke-width", /* appropriate width here */ )
.style("stroke",function(d) { return /* color logic here */ })
This might look something like this (with data mouseover).
This is just a proof of concept, if you were looking for more control and a nicer look, you could create a polygonal path for each data entry and use both fill & stroke. As is, you'll have to make do with layering strokes to get borders if desired and svg manipulations like line capping options.
Also, as it's d3, and longer timespans may be hard to show all at once, you could show less time but rotate the spiral so that it animates through your time span, dropping off events at the end and creating them in the origin. The actual chart might need to be canvas for this to happen smoothly depending on number of nodes, but to convert to canvas is relatively trivial in this case.
For the sake of filling out the visualization a little with a legend and day labels, this is what I have.
I am trying to get the last 8 hrs of data from a dataframe. Is there any way to achieve this?
Suppose I have a dataframe with timestamp:
Time X Y
19/8/2017 10:00:00 824.93 6.42
19/8/2017 10:20:00 825.88 6.41
19/8/2017 10:40:00 825.59 6.45
19/8/2017 11:00:00 826.18 6.6
19/8/2017 11:20:00 825.2 6.62
19/8/2017 11:40:00 825.3 6.65
19/8/2017 12:00:00 824.72 6.54
19/8/2017 12:20:00 825.72 6.45
19/8/2017 12:40:00 824.99 6.35
19/8/2017 13:00:00 824.63 6.31
19/8/2017 13:20:00 825.87 6.34
19/8/2017 13:40:00 826.12 6.29
19/8/2017 14:00:00 825.41 6.31
19/8/2017 14:20:00 825.34 6.36
19/8/2017 14:40:00 825.15 6.39
19/8/2017 15:00:00 824.87 6.15
19/8/2017 15:20:00 825.57 6.17
19/8/2017 15:40:00 825.35 6.2
19/8/2017 16:00:00 824.94 6.26
19/8/2017 16:20:00 825.86 6.39
19/8/2017 16:40:00 824.74 6.39
19/8/2017 17:00:00 823.46 6.33
19/8/2017 17:20:00 824.46 6.41
19/8/2017 17:40:00 824.67 6.4
19/8/2017 18:00:00 824.38 6.36
19/8/2017 18:20:00 825.23 6.37
19/8/2017 18:40:00 826.09 6.37
19/8/2017 19:00:00 824.77 6.32
19/8/2017 19:20:00 824.18 6.34
19/8/2017 19:40:00 823.82 6.33
19/8/2017 20:00:00 825 6.39
19/8/2017 20:20:00 825.01 6.4
19/8/2017 20:40:00 824.24 6.45
19/8/2017 21:00:00 817.65 6.17
19/8/2017 21:20:00 781.58 4.92
19/8/2017 21:40:00 739.89 3.51
I want to extract last 8 hours of data from this dataframe.
I tried this but nothing i can do:
last8 = (df["Time"] == (dfs["Time"].shift() + Timedelta(hours = 8))).cumsum()
Any help would be appreciated.
To get last 8 hours from the max time (last time in this case)
In [2178]: df[df['Time'] > df['Time'].max() - pd.Timedelta(hours=8)]
Out[2178]:
Time X Y
12 2017-08-19 14:00:00 825.41 6.31
13 2017-08-19 14:20:00 825.34 6.36
14 2017-08-19 14:40:00 825.15 6.39
15 2017-08-19 15:00:00 824.87 6.15
16 2017-08-19 15:20:00 825.57 6.17
17 2017-08-19 15:40:00 825.35 6.20
18 2017-08-19 16:00:00 824.94 6.26
19 2017-08-19 16:20:00 825.86 6.39
20 2017-08-19 16:40:00 824.74 6.39
21 2017-08-19 17:00:00 823.46 6.33
22 2017-08-19 17:20:00 824.46 6.41
23 2017-08-19 17:40:00 824.67 6.40
24 2017-08-19 18:00:00 824.38 6.36
25 2017-08-19 18:20:00 825.23 6.37
26 2017-08-19 18:40:00 826.09 6.37
27 2017-08-19 19:00:00 824.77 6.32
28 2017-08-19 19:20:00 824.18 6.34
29 2017-08-19 19:40:00 823.82 6.33
30 2017-08-19 20:00:00 825.00 6.39
31 2017-08-19 20:20:00 825.01 6.40
32 2017-08-19 20:40:00 824.24 6.45
33 2017-08-19 21:00:00 817.65 6.17
34 2017-08-19 21:20:00 781.58 4.92
35 2017-08-19 21:40:00 739.89 3.51
Try this:
df[df.Time >= pd.datetime.now() - pd.Timedelta('8H')]