Similarly to this question, I'm interested in creating time series spirals. The solution doesn't necessarily have to be implemented in R or using ggplot, but it seems the majority of solutions have been implemented in R with ggplot, with a handful in Python and one in d3. My attempts so far have all used R. Unlike this question, I'm interested in displaying specific ranges of data without quantizing/binning the data. That is, I'd like to display a spiral timeline showing when particular events start and stop, where theta-min and theta-max of every event represent specific points in time.
Consider this travel data:
trip_start trip_stop dist
2017-04-01 17:42:00 2017-04-01 18:34:00 1.95
2017-04-01 18:42:00 2017-04-01 19:05:00 6.54
2017-04-02 01:09:00 2017-04-02 01:12:00 1.07
2017-04-02 01:22:00 2017-04-02 01:27:00 1.03
2017-04-02 08:17:00 2017-04-02 08:23:00 1.98
2017-04-02 11:23:00 2017-04-02 11:30:00 1.98
2017-04-02 15:44:00 2017-04-02 15:56:00 4.15
2017-04-02 16:29:00 2017-04-02 16:45:00 4.08
2017-04-03 10:24:00 2017-04-03 10:55:00 19.76
2017-04-03 14:01:00 2017-04-03 14:18:00 8.21
2017-04-03 14:25:00 2017-04-03 14:31:00 1.49
2017-04-03 14:45:00 2017-04-03 14:50:00 1.59
2017-04-03 15:44:00 2017-04-03 16:10:00 4.44
2017-04-03 16:14:00 2017-04-03 16:37:00 9.96
2017-04-03 16:40:00 2017-04-03 16:45:00 0.7
2017-04-03 17:15:00 2017-04-03 17:46:00 16.92
2017-04-03 17:56:00 2017-04-03 18:19:00 5.23
2017-04-03 18:42:00 2017-04-03 18:45:00 0.49
2017-04-03 19:02:00 2017-04-03 19:04:00 0.48
2017-04-04 07:24:00 2017-04-04 07:27:00 0.66
2017-04-04 07:30:00 2017-04-04 08:04:00 13.55
2017-04-04 08:32:00 2017-04-04 09:25:00 25.09
2017-04-04 13:32:00 2017-04-04 13:40:00 3.06
2017-04-04 13:52:00 2017-04-04 13:57:00 1.3
2017-04-04 14:55:00 2017-04-04 15:01:00 2.47
2017-04-04 18:40:00 2017-04-04 19:12:00 22.71
2017-04-04 22:16:00 2017-04-04 23:54:00 38.28
2017-04-04 23:59:00 2017-04-05 00:03:00 1.02
2017-04-05 11:04:00 2017-04-05 11:49:00 25.73
2017-04-05 12:05:00 2017-04-05 12:18:00 2.97
2017-04-05 15:19:00 2017-04-05 16:25:00 25.13
2017-04-05 16:38:00 2017-04-05 16:40:00 0.41
2017-04-05 18:58:00 2017-04-05 19:02:00 1.25
2017-04-05 19:13:00 2017-04-05 19:18:00 1.09
2017-04-05 19:25:00 2017-04-05 19:48:00 6.63
2017-04-06 10:01:00 2017-04-06 10:44:00 20.81
2017-04-06 13:22:00 2017-04-06 13:33:00 1.63
2017-04-06 20:58:00 2017-04-06 21:25:00 24.85
2017-04-06 21:32:00 2017-04-06 21:56:00 6.06
2017-04-07 10:55:00 2017-04-07 11:37:00 24.53
2017-04-07 17:14:00 2017-04-07 17:48:00 19.66
2017-04-07 17:57:00 2017-04-07 18:07:00 2.12
2017-04-08 20:57:00 2017-04-08 21:06:00 1.06
2017-04-08 21:23:00 2017-04-08 21:36:00 2.97
2017-04-09 08:14:00 2017-04-09 08:19:00 1.99
2017-04-09 11:40:00 2017-04-09 11:50:00 2.24
2017-04-09 11:50:00 2017-04-09 11:57:00 1.64
2017-04-09 16:29:00 2017-04-09 16:34:00 0.53
2017-04-09 16:43:00 2017-04-09 16:45:00 0.5
2017-04-09 17:46:00 2017-04-09 17:48:00 0.44
2017-04-09 17:53:00 2017-04-09 17:56:00 0.4
2017-04-09 21:33:00 2017-04-09 21:56:00 2.48
2017-04-09 21:57:00 2017-04-09 22:14:00 2.92
2017-04-09 22:22:00 2017-04-09 22:25:00 0.9
2017-04-10 10:37:00 2017-04-10 11:22:00 19.27
2017-04-10 16:12:00 2017-04-10 16:59:00 21.31
2017-04-11 11:14:00 2017-04-11 11:18:00 1.24
2017-04-11 11:21:00 2017-04-11 11:48:00 22.95
2017-04-11 18:24:00 2017-04-11 19:05:00 28.64
2017-04-11 19:21:00 2017-04-11 19:34:00 5.37
2017-04-12 11:00:00 2017-04-12 12:08:00 28.91
2017-04-12 14:03:00 2017-04-12 15:20:00 28.56
2017-04-12 20:24:00 2017-04-12 20:29:00 1.17
2017-04-12 20:32:00 2017-04-12 21:09:00 30.89
2017-04-13 01:37:00 2017-04-13 02:09:00 32.3
2017-04-13 08:08:00 2017-04-13 08:39:00 19.39
2017-04-13 10:53:00 2017-04-13 11:23:00 24.59
2017-04-13 18:56:00 2017-04-13 19:22:00 22.74
2017-04-14 01:06:00 2017-04-14 01:37:00 31.36
2017-04-14 01:48:00 2017-04-14 01:51:00 1.03
2017-04-14 12:08:00 2017-04-14 12:22:00 1.94
2017-04-14 12:29:00 2017-04-14 13:01:00 19.07
2017-04-14 16:17:00 2017-04-14 17:03:00 19.74
2017-04-14 17:05:00 2017-04-14 17:32:00 3.99
2017-04-14 21:57:00 2017-04-14 22:02:00 1.98
2017-04-15 01:46:00 2017-04-15 01:49:00 1.07
2017-04-15 01:56:00 2017-04-15 01:58:00 1.03
2017-04-15 07:13:00 2017-04-15 07:15:00 0.45
2017-04-15 07:19:00 2017-04-15 07:21:00 0.41
2017-04-15 15:54:00 2017-04-15 16:05:00 1.94
2017-04-15 22:23:00 2017-04-15 22:26:00 0.86
2017-04-15 22:46:00 2017-04-15 22:47:00 0.25
2017-04-15 22:51:00 2017-04-15 22:53:00 0.71
2017-04-16 11:35:00 2017-04-16 11:54:00 11.4
2017-04-16 11:58:00 2017-04-16 12:15:00 10.43
2017-04-17 10:44:00 2017-04-17 10:53:00 3.04
2017-04-17 10:55:00 2017-04-17 11:22:00 18.26
2017-04-17 18:09:00 2017-04-17 18:12:00 0.85
2017-04-17 18:21:00 2017-04-17 19:07:00 37.22
2017-04-18 02:07:00 2017-04-18 02:47:00 32.41
2017-04-18 10:55:00 2017-04-18 10:57:00 0.41
2017-04-18 11:02:00 2017-04-18 11:12:00 2.3
2017-04-18 11:15:00 2017-04-18 11:52:00 24.05
2017-04-18 16:59:00 2017-04-18 17:55:00 22.66
2017-04-19 00:46:00 2017-04-19 01:35:00 39.25
2017-04-19 10:57:00 2017-04-19 11:44:00 24.06
2017-04-19 13:23:00 2017-04-19 14:10:00 25.96
2017-04-19 16:21:00 2017-04-19 17:07:00 18.05
2017-04-19 23:32:00 2017-04-20 00:19:00 39.67
2017-04-20 10:47:00 2017-04-20 11:13:00 24.07
2017-04-20 16:21:00 2017-04-20 16:30:00 0.86
2017-04-20 16:36:00 2017-04-20 16:58:00 0.85
2017-04-20 17:41:00 2017-04-20 17:44:00 0.37
2017-04-20 17:49:00 2017-04-20 18:40:00 19.32
2017-04-20 22:22:00 2017-04-20 22:53:00 29.2
2017-04-20 23:07:00 2017-04-20 23:27:00 10.94
2017-04-21 08:29:00 2017-04-21 08:40:00 1.91
2017-04-21 11:30:00 2017-04-21 11:32:00 0.42
2017-04-21 11:38:00 2017-04-21 11:40:00 0.4
2017-04-21 11:42:00 2017-04-21 12:15:00 19.09
2017-04-21 16:50:00 2017-04-21 18:17:00 40.61
2017-04-21 18:55:00 2017-04-21 19:11:00 1.73
2017-04-21 22:20:00 2017-04-21 22:53:00 28.26
2017-04-21 23:01:00 2017-04-21 23:22:00 11.76
2017-04-22 08:56:00 2017-04-22 08:58:00 0.63
2017-04-22 09:04:00 2017-04-22 09:08:00 0.3
2017-04-22 09:12:00 2017-04-22 09:15:00 0.42
2017-04-22 16:48:00 2017-04-22 16:52:00 0.54
2017-04-22 17:06:00 2017-04-22 17:09:00 0.51
2017-04-22 17:10:00 2017-04-22 17:13:00 1.03
2017-04-22 17:22:00 2017-04-22 17:27:00 1.1
2017-04-23 08:13:00 2017-04-23 08:15:00 0.41
2017-04-23 08:19:00 2017-04-23 08:20:00 0.4
2017-04-23 08:21:00 2017-04-23 08:25:00 1.99
2017-04-23 11:41:00 2017-04-23 11:48:00 2.04
2017-04-23 12:35:00 2017-04-23 12:50:00 7.59
2017-04-23 14:08:00 2017-04-23 14:21:00 7.31
2017-04-23 14:33:00 2017-04-23 15:38:00 37.6
2017-04-24 00:26:00 2017-04-24 01:18:00 39.21
2017-04-24 10:24:00 2017-04-24 10:26:00 0.41
2017-04-24 10:31:00 2017-04-24 10:35:00 1.37
2017-04-24 10:38:00 2017-04-24 10:43:00 1.19
2017-04-24 10:49:00 2017-04-24 11:15:00 19.58
2017-04-24 17:13:00 2017-04-24 18:20:00 37.42
2017-04-24 19:02:00 2017-04-24 19:08:00 1.76
2017-04-24 19:49:00 2017-04-24 19:55:00 1.79
2017-04-24 20:41:00 2017-04-24 21:16:00 32.31
2017-04-25 10:53:00 2017-04-25 11:25:00 24.83
2017-04-25 15:15:00 2017-04-25 15:24:00 3.07
2017-04-25 15:30:00 2017-04-25 15:40:00 3.01
2017-04-25 17:34:00 2017-04-25 18:18:00 24.8
2017-04-26 09:59:00 2017-04-26 10:28:00 24.05
2017-04-26 12:56:00 2017-04-26 13:40:00 29.13
2017-04-26 14:37:00 2017-04-26 15:34:00 21
2017-04-27 08:57:00 2017-04-27 10:21:00 40.56
2017-04-27 16:12:00 2017-04-27 16:44:00 9.89
2017-04-27 17:09:00 2017-04-27 18:01:00 17.51
2017-04-28 05:18:00 2017-04-28 06:06:00 39.28
2017-04-28 12:57:00 2017-04-28 13:52:00 35.82
2017-04-28 16:48:00 2017-04-28 18:14:00 39.1
2017-05-01 11:41:00 2017-05-01 12:20:00 18.74
2017-05-01 18:53:00 2017-05-01 19:34:00 37.15
2017-05-01 23:08:00 2017-05-01 23:09:00 0.06
2017-05-01 23:18:00 2017-05-02 00:11:00 38.61
2017-05-02 11:05:00 2017-05-02 11:42:00 24.07
2017-05-02 17:34:00 2017-05-02 18:53:00 26.42
2017-05-03 12:13:00 2017-05-03 12:25:00 3.96
2017-05-03 12:25:00 2017-05-03 12:56:00 21.15
2017-05-03 13:26:00 2017-05-03 13:44:00 3.32
2017-05-03 13:57:00 2017-05-03 14:08:00 3.49
2017-05-03 18:39:00 2017-05-03 19:08:00 24.85
2017-05-03 19:09:00 2017-05-03 19:13:00 0.99
2017-05-03 19:29:00 2017-05-03 19:32:00 0.84
2017-05-04 10:38:00 2017-05-04 11:06:00 24.05
2017-05-04 13:34:00 2017-05-04 14:10:00 1.73
2017-05-04 17:14:00 2017-05-04 18:23:00 24.68
2017-05-05 20:38:00 2017-05-05 20:52:00 2.24
2017-05-06 11:45:00 2017-05-06 12:30:00 20.19
2017-05-06 14:36:00 2017-05-06 15:35:00 14.49
2017-05-06 15:48:00 2017-05-06 16:17:00 5.25
2017-05-06 17:11:00 2017-05-06 17:13:00 0.43
2017-05-06 17:19:00 2017-05-06 17:21:00 0.43
2017-05-07 08:16:00 2017-05-07 08:22:00 3.27
2017-05-07 12:09:00 2017-05-07 12:16:00 2.01
2017-05-07 17:28:00 2017-05-07 17:50:00 10.36
2017-05-07 17:54:00 2017-05-07 18:01:00 1.19
2017-05-07 18:02:00 2017-05-07 18:35:00 28.31
2017-05-07 21:48:00 2017-05-07 21:52:00 1.46
2017-05-07 22:01:00 2017-05-07 22:05:00 1.37
2017-05-08 00:59:00 2017-05-08 02:19:00 39.23
2017-05-08 11:30:00 2017-05-08 11:58:00 22.55
2017-05-08 18:08:00 2017-05-08 18:30:00 10.47
2017-05-08 18:33:00 2017-05-08 19:09:00 28.44
2017-05-08 22:25:00 2017-05-08 23:09:00 38.65
2017-05-08 23:14:00 2017-05-08 23:17:00 1.04
2017-05-09 11:35:00 2017-05-09 12:19:00 23.99
2017-05-09 17:57:00 2017-05-09 18:59:00 29.38
2017-05-09 20:03:00 2017-05-09 20:13:00 1.9
2017-05-10 10:18:00 2017-05-10 10:54:00 24.06
2017-05-10 15:43:00 2017-05-10 16:46:00 24.71
2017-05-11 12:28:00 2017-05-11 13:07:00 21.75
2017-05-11 18:00:00 2017-05-11 18:31:00 19.3
2017-05-12 08:26:00 2017-05-12 08:55:00 20.46
2017-05-12 13:00:00 2017-05-12 13:34:00 14.6
2017-05-13 08:44:00 2017-05-13 08:46:00 0.38
2017-05-13 08:57:00 2017-05-13 09:01:00 0.33
2017-05-13 14:22:00 2017-05-13 14:41:00 6.86
2017-05-13 15:17:00 2017-05-13 15:35:00 5.2
2017-05-13 18:10:00 2017-05-13 18:21:00 1.91
2017-05-14 11:22:00 2017-05-14 11:26:00 0.9
2017-05-14 11:36:00 2017-05-14 11:38:00 0.39
2017-05-14 14:56:00 2017-05-14 15:59:00 40.07
2017-05-14 16:34:00 2017-05-14 16:41:00 1.49
2017-05-14 16:56:00 2017-05-14 17:04:00 1.45
2017-05-14 19:05:00 2017-05-14 20:06:00 39.21
2017-05-15 11:24:00 2017-05-15 11:33:00 1.91
2017-05-15 11:41:00 2017-05-15 12:13:00 19.84
2017-05-15 17:41:00 2017-05-15 18:11:00 16
2017-05-15 18:15:00 2017-05-15 19:23:00 31.52
2017-05-15 23:41:00 2017-05-16 00:26:00 39.32
2017-05-16 09:49:00 2017-05-16 11:02:00 24.91
2017-05-16 16:08:00 2017-05-16 16:32:00 3.37
2017-05-16 17:11:00 2017-05-16 17:32:00 4.8
2017-05-16 17:42:00 2017-05-16 17:56:00 1.81
2017-05-16 18:13:00 2017-05-16 18:46:00 24.85
2017-05-16 21:07:00 2017-05-16 21:10:00 1.04
2017-05-16 21:26:00 2017-05-16 21:29:00 1.02
2017-07-28 16:10:00 2017-07-28 16:17:00 2.22
2017-07-28 16:17:00 2017-07-28 16:42:00 7.84
2017-08-10 12:00:00 2017-08-10 12:44:00 24.05
2017-08-10 14:56:00 2017-08-10 15:10:00 1.61
2017-08-10 18:51:00 2017-08-10 19:21:00 24.85
2017-08-10 19:46:00 2017-08-10 19:56:00 1.14
2017-08-10 20:08:00 2017-08-10 20:12:00 1.09
2017-08-11 12:44:00 2017-08-11 12:49:00 0.82
2017-08-11 12:59:00 2017-08-11 13:01:00 0.56
2017-08-11 13:18:00 2017-08-11 15:12:00 1.79
2017-08-11 15:14:00 2017-08-11 16:53:00 34.6
2017-08-11 19:27:00 2017-08-11 20:34:00 34.91
2017-08-12 13:52:00 2017-08-12 13:56:00 1.05
2017-08-12 13:59:00 2017-08-12 14:02:00 0.28
2017-08-12 14:10:00 2017-08-12 14:30:00 1.22
2017-08-12 17:15:00 2017-08-12 17:36:00 11.37
2017-08-12 20:49:00 2017-08-12 21:05:00 10.43
2017-08-13 12:16:00 2017-08-13 12:44:00 12.96
2017-08-13 16:03:00 2017-08-13 16:32:00 14.33
2017-08-13 18:19:00 2017-08-13 18:42:00 9.32
2017-08-13 18:52:00 2017-08-13 19:05:00 3.99
2017-08-13 21:42:00 2017-08-13 21:53:00 5.6
2017-08-14 08:50:00 2017-08-14 09:45:00 24.1
2017-08-14 13:22:00 2017-08-14 13:54:00 24.84
2017-08-14 14:02:00 2017-08-14 15:34:00 36.92
2017-08-14 15:58:00 2017-08-14 17:17:00 35.7
2017-08-14 17:35:00 2017-08-14 17:45:00 1.99
2017-08-14 18:07:00 2017-08-14 18:27:00 9.92
2017-08-15 10:15:00 2017-08-15 10:51:00 25
2017-08-15 19:23:00 2017-08-15 19:29:00 0.4
2017-08-15 19:51:00 2017-08-15 20:45:00 24.39
2017-08-15 20:56:00 2017-08-15 21:04:00 2.78
2017-08-15 21:09:00 2017-08-15 21:37:00 19.22
2017-08-16 00:03:00 2017-08-16 00:27:00 15.51
2017-08-16 00:36:00 2017-08-16 00:41:00 1.23
2017-08-16 00:46:00 2017-08-16 01:18:00 11.35
2017-08-16 09:38:00 2017-08-16 09:41:00 1.21
2017-08-16 09:41:00 2017-08-16 09:43:00 0.08
2017-08-16 09:47:00 2017-08-16 10:32:00 22.89
2017-08-16 16:51:00 2017-08-16 17:11:00 3.14
2017-08-16 17:12:00 2017-08-16 17:25:00 2.76
2017-08-16 17:41:00 2017-08-16 18:36:00 24.78
2017-08-17 09:34:00 2017-08-17 10:13:00 24.03
2017-08-17 12:32:00 2017-08-17 13:07:00 24.82
2017-08-17 13:35:00 2017-08-17 13:40:00 0.4
2017-08-17 13:47:00 2017-08-17 15:07:00 36.06
2017-08-17 15:18:00 2017-08-17 15:24:00 0.06
2017-08-17 16:03:00 2017-08-17 18:05:00 35.16
2017-08-18 09:47:00 2017-08-18 10:23:00 24.47
2017-08-18 16:04:00 2017-08-18 16:42:00 1.63
2017-08-18 17:56:00 2017-08-18 18:25:00 10.74
2017-08-18 18:27:00 2017-08-18 18:48:00 1.85
2017-08-19 00:07:00 2017-08-19 00:41:00 18.92
2017-08-19 00:52:00 2017-08-19 00:55:00 0.99
2017-08-19 11:52:00 2017-08-19 12:14:00 7.56
2017-08-19 15:57:00 2017-08-19 16:12:00 4.02
2017-08-19 16:37:00 2017-08-19 16:56:00 5.32
2017-08-19 23:32:00 2017-08-19 23:50:00 7.54
2017-08-19 23:51:00 2017-08-20 00:17:00 9.59
2017-08-20 09:03:00 2017-08-20 09:16:00 5.22
2017-08-20 19:17:00 2017-08-20 19:32:00 4.69
2017-08-21 09:24:00 2017-08-21 09:40:00 2.31
2017-08-21 10:59:00 2017-08-21 11:02:00 0.47
2017-08-21 13:40:00 2017-08-21 15:29:00 36.09
2017-08-21 15:54:00 2017-08-21 16:48:00 2.24
2017-08-21 16:57:00 2017-08-21 18:15:00 32.3
2017-08-22 08:38:00 2017-08-22 09:06:00 0.65
2017-08-22 09:18:00 2017-08-22 09:19:00 0.04
2017-08-22 09:22:00 2017-08-22 10:05:00 23.49
2017-08-22 14:30:00 2017-08-22 15:02:00 1.7
2017-08-22 16:37:00 2017-08-22 17:41:00 24.8
2017-08-23 17:16:00 2017-08-23 18:14:00 24.01
2017-08-23 18:27:00 2017-08-23 18:32:00 1.05
2017-08-23 19:24:00 2017-08-23 20:04:00 18.14
2017-08-23 22:01:00 2017-08-23 22:28:00 16.33
2017-08-23 22:46:00 2017-08-23 22:50:00 1.04
2017-08-24 09:41:00 2017-08-24 09:44:00 0.02
2017-08-24 09:59:00 2017-08-24 10:00:00 0.02
2017-08-24 13:57:00 2017-08-24 15:33:00 42.51
2017-08-24 16:43:00 2017-08-24 17:00:00 0.07
2017-08-24 17:06:00 2017-08-24 17:33:00 10.01
2017-08-24 18:12:00 2017-08-24 19:03:00 27.67
2017-08-25 09:36:00 2017-08-25 09:55:00 2.63
2017-08-25 10:01:00 2017-08-25 10:32:00 20.92
2017-08-25 20:40:00 2017-08-25 21:45:00 17.41
2017-08-25 21:49:00 2017-08-25 22:14:00 16.02
2017-08-26 00:10:00 2017-08-26 02:14:00 29.77
2017-08-26 16:31:00 2017-08-26 16:55:00 7.15
2017-08-26 17:54:00 2017-08-26 18:19:00 10
2017-08-26 20:07:00 2017-08-26 20:08:00 0.19
2017-08-26 20:08:00 2017-08-26 20:11:00 1.35
2017-08-27 12:39:00 2017-08-27 12:54:00 1
2017-08-27 12:55:00 2017-08-27 13:48:00 9.29
2017-08-27 14:00:00 2017-08-27 14:34:00 3.86
2017-08-27 15:56:00 2017-08-27 16:37:00 10.45
2017-08-27 16:44:00 2017-08-27 16:51:00 1.8
2017-08-27 16:55:00 2017-08-27 17:00:00 0.68
2017-08-27 17:04:00 2017-08-27 17:19:00 4.96
2017-08-27 17:28:00 2017-08-27 17:39:00 2.33
2017-08-27 17:47:00 2017-08-27 18:58:00 24.19
2017-08-27 22:17:00 2017-08-27 22:41:00 16.24
2017-08-28 00:33:00 2017-08-28 01:22:00 13.62
2017-08-28 12:48:00 2017-08-28 12:51:00 0.47
2017-08-28 14:01:00 2017-08-28 14:03:00 0.4
2017-08-28 14:12:00 2017-08-28 15:31:00 34.86
2017-08-28 15:56:00 2017-08-28 17:04:00 34.47
2017-08-28 22:15:00 2017-08-28 22:38:00 18.57
2017-08-29 01:42:00 2017-08-29 02:05:00 18.88
2017-08-29 11:40:00 2017-08-29 11:44:00 1.04
2017-08-29 11:48:00 2017-08-29 12:09:00 0.03
2017-08-29 12:18:00 2017-08-29 12:21:00 0.03
2017-08-29 12:26:00 2017-08-29 12:32:00 1.05
2017-08-29 12:35:00 2017-08-29 13:15:00 24.05
2017-08-29 19:40:00 2017-08-29 19:42:00 0.35
2017-08-29 19:50:00 2017-08-29 20:19:00 27.72
2017-08-29 20:25:00 2017-08-29 20:41:00 10.42
2017-08-30 10:00:00 2017-08-30 10:47:00 24.25
2017-08-30 14:31:00 2017-08-30 14:56:00 1.68
2017-08-30 17:19:00 2017-08-30 17:43:00 0.04
2017-08-30 17:43:00 2017-08-30 17:50:00 0.29
2017-08-30 17:56:00 2017-08-30 18:40:00 16.85
2017-08-30 22:57:00 2017-08-30 23:35:00 17.31
2017-08-31 11:30:00 2017-08-31 11:41:00 0.43
2017-08-31 14:04:00 2017-08-31 14:06:00 0.41
2017-08-31 14:24:00 2017-08-31 14:26:00 0.68
2017-08-31 14:31:00 2017-08-31 15:42:00 34.88
2017-08-31 16:01:00 2017-08-31 17:07:00 30.45
2017-08-31 20:54:00 2017-08-31 21:21:00 19.6
2017-09-01 10:30:00 2017-09-01 10:59:00 17.63
2017-09-01 14:07:00 2017-09-01 15:07:00 27.45
2017-09-01 17:17:00 2017-09-01 17:36:00 1.93
2017-09-01 18:16:00 2017-09-01 19:19:00 20.58
2017-09-01 19:25:00 2017-09-01 19:38:00 4.8
2017-09-01 21:30:00 2017-09-01 21:54:00 1.94
2017-09-02 15:46:00 2017-09-02 16:06:00 0.99
2017-09-02 16:13:00 2017-09-02 16:16:00 1.01
2017-09-02 16:56:00 2017-09-02 16:59:00 0.42
2017-09-02 17:04:00 2017-09-02 17:06:00 0.4
2017-09-02 22:52:00 2017-09-02 22:54:00 0.07
2017-09-02 22:55:00 2017-09-02 23:15:00 18.62
2017-09-03 01:46:00 2017-09-03 02:10:00 18.9
2017-09-03 14:49:00 2017-09-03 15:04:00 3.14
2017-09-03 15:50:00 2017-09-03 16:07:00 10.17
2017-09-03 16:21:00 2017-09-03 16:38:00 7.79
2017-09-03 16:47:00 2017-09-03 16:52:00 1.11
2017-09-03 18:32:00 2017-09-03 18:37:00 1.2
2017-09-03 18:37:00 2017-09-03 18:44:00 0.91
2017-09-04 15:50:00 2017-09-04 15:54:00 0.42
2017-09-04 15:59:00 2017-09-04 16:11:00 2.3
2017-09-04 16:21:00 2017-09-04 16:43:00 8.31
2017-09-04 17:05:00 2017-09-04 17:15:00 2.54
2017-09-04 17:26:00 2017-09-04 17:41:00 4.52
2017-09-04 17:49:00 2017-09-04 18:25:00 29.55
2017-09-04 19:36:00 2017-09-04 19:51:00 0.93
2017-09-04 19:54:00 2017-09-04 19:59:00 0.5
2017-09-04 21:21:00 2017-09-04 21:55:00 29.37
2017-09-05 11:08:00 2017-09-05 11:51:00 35.5
2017-09-05 12:36:00 2017-09-05 13:07:00 2.29
2017-09-05 13:19:00 2017-09-05 13:22:00 0.51
2017-09-05 13:26:00 2017-09-05 14:03:00 33.09
2017-09-05 14:13:00 2017-09-05 15:01:00 24.03
2017-09-05 17:33:00 2017-09-05 18:11:00 14.55
2017-09-05 19:01:00 2017-09-05 19:19:00 11.31
2017-09-06 09:21:00 2017-09-06 09:39:00 7.73
2017-09-06 10:14:00 2017-09-06 10:30:00 7.75
2017-09-06 10:37:00 2017-09-06 11:13:00 24.13
2017-09-06 16:48:00 2017-09-06 17:35:00 25.3
2017-09-06 17:49:00 2017-09-06 17:55:00 0.18
2017-09-06 17:58:00 2017-09-06 18:00:00 0.39
2017-09-06 18:38:00 2017-09-06 19:04:00 15.93
2017-09-06 23:45:00 2017-09-07 00:14:00 19.45
2017-09-07 00:26:00 2017-09-07 00:30:00 1.01
2017-09-07 10:42:00 2017-09-07 11:35:00 31.74
2017-09-07 14:04:00 2017-09-07 14:39:00 27.38
2017-09-07 14:43:00 2017-09-07 14:52:00 3.06
2017-09-07 14:54:00 2017-09-07 16:00:00 32.96
2017-09-07 16:32:00 2017-09-07 16:33:00 0.07
2017-09-07 16:38:00 2017-09-07 17:04:00 2.31
2017-09-07 17:23:00 2017-09-07 18:14:00 33.03
2017-09-08 10:02:00 2017-09-08 10:30:00 19.73
2017-09-08 18:09:00 2017-09-08 18:37:00 18.97
2017-09-08 19:04:00 2017-09-08 19:18:00 1.87
2017-09-09 02:25:00 2017-09-09 02:28:00 1.1
2017-09-09 02:33:00 2017-09-09 02:35:00 1.05
2017-09-10 17:09:00 2017-09-10 17:44:00 14.25
2017-09-10 22:50:00 2017-09-10 22:53:00 0.25
2017-09-10 22:56:00 2017-09-10 22:57:00 0.02
2017-09-10 23:00:00 2017-09-10 23:23:00 16.18
2017-09-11 00:01:00 2017-09-11 00:19:00 1.83
2017-09-11 09:59:00 2017-09-11 10:06:00 1.91
2017-09-11 10:12:00 2017-09-11 10:51:00 27.49
2017-09-11 13:39:00 2017-09-11 14:13:00 27.23
2017-09-11 14:31:00 2017-09-11 15:31:00 35.45
2017-09-11 16:03:00 2017-09-11 17:09:00 36.01
2017-09-11 17:39:00 2017-09-11 18:01:00 9.88
2017-09-11 23:01:00 2017-09-11 23:05:00 1.14
2017-09-11 23:16:00 2017-09-11 23:30:00 5.93
2017-09-11 23:30:00 2017-09-11 23:54:00 4.94
2017-09-12 02:56:00 2017-09-12 04:00:00 25.87
2017-09-12 10:06:00 2017-09-12 10:46:00 24.84
2017-09-12 16:33:00 2017-09-12 17:20:00 22.43
2017-09-12 19:38:00 2017-09-12 20:14:00 21.79
2017-09-13 06:24:00 2017-09-13 06:59:00 25.84
2017-09-13 07:02:00 2017-09-13 07:14:00 5.77
2017-09-13 11:14:00 2017-09-13 11:36:00 16.26
2017-09-13 16:01:00 2017-09-13 16:57:00 24.79
2017-09-13 17:07:00 2017-09-13 17:48:00 15.94
2017-09-13 23:13:00 2017-09-13 23:35:00 16.73
2017-09-14 12:00:00 2017-09-14 12:27:00 19.71
2017-09-14 12:28:00 2017-09-14 12:30:00 0.18
2017-09-14 14:36:00 2017-09-14 15:06:00 14.98
2017-09-14 15:11:00 2017-09-14 15:17:00 2.99
2017-09-14 15:26:00 2017-09-14 16:44:00 37.48
2017-09-14 17:03:00 2017-09-14 18:17:00 34.18
2017-09-14 18:32:00 2017-09-14 18:41:00 3.03
2017-09-15 10:25:00 2017-09-15 10:26:00 0.05
2017-09-15 10:45:00 2017-09-15 10:48:00 0.29
2017-09-15 10:59:00 2017-09-15 11:05:00 0.3
2017-09-15 11:09:00 2017-09-15 11:36:00 10.82
2017-09-15 13:00:00 2017-09-15 13:17:00 8.37
2017-09-15 13:36:00 2017-09-15 14:30:00 25.19
2017-09-15 14:37:00 2017-09-15 15:01:00 0.45
2017-09-15 15:04:00 2017-09-15 16:59:00 85.51
2017-09-15 17:06:00 2017-09-15 18:57:00 129.72
2017-09-15 19:03:00 2017-09-15 20:02:00 60.96
2017-09-16 10:18:00 2017-09-16 10:39:00 16.04
2017-09-16 11:52:00 2017-09-16 12:12:00 16.68
2017-09-16 12:28:00 2017-09-16 13:29:00 49
2017-09-16 18:36:00 2017-09-16 19:30:00 45.7
2017-09-16 19:39:00 2017-09-16 19:47:00 2.1
2017-09-17 13:32:00 2017-09-17 13:41:00 2.24
2017-09-17 14:19:00 2017-09-17 14:48:00 14.68
2017-09-17 18:25:00 2017-09-17 18:26:00 0.05
2017-09-17 18:36:00 2017-09-17 19:03:00 12.26
2017-09-18 07:52:00 2017-09-18 08:03:00 2.04
2017-09-18 08:21:00 2017-09-18 08:56:00 37.94
2017-09-18 09:01:00 2017-09-18 09:53:00 65.7
2017-09-18 10:04:00 2017-09-18 10:34:00 39.43
2017-09-18 10:46:00 2017-09-18 11:07:00 14.25
2017-09-18 11:19:00 2017-09-18 13:29:00 138.98
2017-09-18 14:24:00 2017-09-18 14:26:00 0.04
2017-09-18 14:28:00 2017-09-18 15:23:00 35.52
2017-09-18 15:53:00 2017-09-18 17:49:00 36.64
2017-09-19 09:24:00 2017-09-19 10:22:00 24.37
2017-09-19 15:55:00 2017-09-19 16:53:00 15.87
2017-09-19 16:53:00 2017-09-19 17:20:00 0.85
2017-09-19 17:33:00 2017-09-19 18:06:00 10.95
2017-09-19 18:10:00 2017-09-19 18:34:00 8.41
2017-09-19 21:06:00 2017-09-19 21:10:00 1.24
2017-09-19 21:17:00 2017-09-19 21:21:00 1.05
2017-09-20 11:12:00 2017-09-20 11:16:00 1.22
2017-09-20 11:18:00 2017-09-20 11:59:00 24.15
2017-09-20 17:20:00 2017-09-20 18:07:00 24.15
2017-09-20 18:50:00 2017-09-20 19:17:00 16.02
2017-09-20 22:05:00 2017-09-20 22:32:00 17.5
2017-09-21 13:38:00 2017-09-21 13:44:00 0.72
2017-09-21 13:50:00 2017-09-21 15:26:00 35.81
2017-09-21 15:59:00 2017-09-21 16:15:00 8.26
2017-09-21 16:19:00 2017-09-21 17:32:00 28.1
2017-09-21 18:49:00 2017-09-21 19:25:00 16.05
2017-09-21 22:30:00 2017-09-21 22:59:00 16.97
2017-09-22 10:19:00 2017-09-22 10:21:00 0.43
2017-09-22 10:25:00 2017-09-22 10:26:00 0.4
2017-09-22 10:30:00 2017-09-22 10:54:00 19.15
2017-09-22 11:58:00 2017-09-22 12:02:00 1.05
2017-09-22 18:32:00 2017-09-22 18:59:00 20.95
2017-09-23 08:34:00 2017-09-23 08:51:00 1.15
2017-09-23 09:19:00 2017-09-23 10:31:00 37.57
2017-09-23 11:09:00 2017-09-23 11:23:00 5.67
2017-09-23 11:51:00 2017-09-23 12:15:00 4.64
2017-09-23 12:47:00 2017-09-23 13:40:00 8.45
2017-09-23 13:56:00 2017-09-23 15:08:00 34.62
2017-09-23 15:37:00 2017-09-23 16:07:00 1.56
2017-09-24 14:59:00 2017-09-24 15:02:00 0.43
2017-09-24 15:14:00 2017-09-24 17:09:00 6.6
2017-09-24 17:37:00 2017-09-24 18:01:00 7.05
2017-09-24 18:05:00 2017-09-24 18:07:00 0.41
2017-09-24 19:35:00 2017-09-24 20:31:00 25.28
2017-09-25 00:24:00 2017-09-25 00:26:00 0.42
2017-09-25 00:30:00 2017-09-25 01:10:00 23.13
2017-09-25 12:12:00 2017-09-25 12:38:00 19.45
2017-09-25 14:22:00 2017-09-25 14:50:00 19.86
2017-09-25 14:52:00 2017-09-25 15:54:00 35.53
2017-09-25 16:37:00 2017-09-25 18:17:00 34.54
2017-09-25 20:36:00 2017-09-25 21:08:00 28.91
2017-09-26 01:46:00 2017-09-26 02:21:00 26.32
2017-09-26 09:36:00 2017-09-26 10:18:00 24.02
2017-09-26 14:05:00 2017-09-26 14:39:00 25.3
2017-09-26 15:49:00 2017-09-26 15:58:00 1.53
2017-09-26 16:15:00 2017-09-26 16:22:00 1.1
2017-09-27 09:15:00 2017-09-27 10:16:00 24.76
2017-09-27 16:26:00 2017-09-27 17:49:00 35.87
2017-09-27 17:58:00 2017-09-27 18:46:00 27.64
2017-09-27 18:51:00 2017-09-27 18:59:00 2.08
2017-09-27 19:10:00 2017-09-27 20:17:00 21.17
2017-09-27 20:25:00 2017-09-27 21:56:00 3.6
2017-09-27 22:04:00 2017-09-27 22:32:00 16.56
2017-09-28 06:46:00 2017-09-28 07:19:00 14.4
2017-09-28 09:05:00 2017-09-28 09:29:00 8.06
2017-09-28 10:41:00 2017-09-28 11:21:00 22.34
2017-09-28 14:26:00 2017-09-28 16:05:00 35.57
2017-09-28 16:09:00 2017-09-28 16:21:00 1.17
2017-09-28 20:37:00 2017-09-28 20:40:00 1.1
2017-09-28 20:56:00 2017-09-28 21:00:00 1.15
2017-09-29 09:32:00 2017-09-29 10:02:00 19.73
I'd like to plot these discrete events the same way the below plots do, but where 2pi is one week rather than 24 hours in order to illuminate the periodicity of these events, where color represents distance.
I've attempted modifying the solution linked at the beginning of this question, but it hasn't gotten me anywhere. My new approach is to modify this solution, but I'm having a difficult time getting anything but horizontal and vertical lines scattered about a spiral. Making them curve and display in the correct locations is tough.
I'm open to any approach that successfully displays the data in a spiral plot without quantizing/binning it into specific intervals but rather allows the intervals themselves to describe discrete events along a continuous spiralling timeline. Likewise, I'm not interested in converting this to a raw single-point time series format where I'd have a great deal of data representing the time between trips. I'd like to achieve this in a temporal format (one that describes a time window rather than an event at a particular time).
Still needs work, but it's a start, with python and matplotlib.
The idea is to plot a spiral timeline in polar coordinates with 1 week period, each event is an arc of this spiral with a color depending on dist data.
There are lots of overlapping intervals though that this visualization tends to hide... maybe semitransparent arcs could be better, with a carefully chosen colormap.
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.patheffects as mpe
import pandas as pd
# styling
LINEWIDTH=4
EDGEWIDTH=1
CAPSTYLE="projecting"
COLORMAP="viridis_r"
ALPHA=1
FIRSTDAY=6 # 0=Mon, 6=Sun
# load dataset and parse timestamps
df = pd.read_csv('trips.csv')
df[['trip_start', 'trip_stop']] = df[['trip_start', 'trip_stop']].apply(pd.to_datetime)
# set origin at the first FIRSTDAY before the first trip, midnight
first_trip = df['trip_start'].min()
origin = (first_trip - pd.to_timedelta(first_trip.weekday() - FIRSTDAY, unit='d')).replace(hour=0, minute=0, second=0)
weekdays = pd.date_range(origin, origin + np.timedelta64(1, 'W')).strftime("%a").tolist()
# # convert trip timestamps to week fractions
df['start'] = (df['trip_start'] - origin) / np.timedelta64(1, 'W')
df['stop'] = (df['trip_stop'] - origin) / np.timedelta64(1, 'W')
# sort dataset so shortest trips are plotted last
# should prevent longer events to cover shorter ones, still suboptimal
df = df.sort_values('dist', ascending=False).reset_index()
fig = plt.figure(figsize=(8, 6))
ax = fig.gca(projection="polar")
for idx, event in df.iterrows():
# sample normalized distance from colormap
ndist = event['dist'] / df['dist'].max()
color = plt.cm.get_cmap(COLORMAP)(ndist)
tstart, tstop = event.loc[['start', 'stop']]
# timestamps are in week fractions, 2pi is one week
nsamples = int(1000. * (tstop - tstart))
t = np.linspace(tstart, tstop, nsamples)
theta = 2 * np.pi * t
arc, = ax.plot(theta, t, lw=LINEWIDTH, color=color, solid_capstyle=CAPSTYLE, alpha=ALPHA)
if EDGEWIDTH > 0:
arc.set_path_effects([mpe.Stroke(linewidth=LINEWIDTH+EDGEWIDTH, foreground='black'), mpe.Normal()])
# grid and labels
ax.set_rticks([])
ax.set_theta_zero_location("N")
ax.set_theta_direction(-1)
ax.set_xticks(np.linspace(0, 2*np.pi, 7, endpoint=False))
ax.set_xticklabels(weekdays)
ax.tick_params('x', pad=2)
ax.grid(True)
# setup a custom colorbar, everything's always a bit tricky with mpl colorbars
vmin = df['dist'].min()
vmax = df['dist'].max()
norm = mpl.colors.Normalize(vmin=vmin, vmax=vmax)
sm = plt.cm.ScalarMappable(cmap=COLORMAP, norm=norm)
sm.set_array([])
plt.colorbar(sm, ticks=np.linspace(vmin, vmax, 10), fraction=0.04, aspect=60, pad=0.1, label="distance", ax=ax)
plt.savefig("spiral.png", pad_inches=0, bbox_inches="tight")
Full timeline
To see it's a spiral that never overlaps and it works for longer events too you can plot the full timeline (here with LINEWIDTH=3.5 to limit moiré fringing).
fullt = np.linspace(df['start'].min(), df['stop'].max(), 10000)
theta = 2 * np.pi * fullt
ax.plot(theta, fullt, lw=LINEWIDTH,
path_effects=[mpe.Stroke(linewidth=LINEWIDTH+LINEBORDER, foreground='black'), mpe.Normal()])
Example with a random set...
Here's the plot for a random dataset of 200 mainly short trips with the occasional 1 to 2 weeks long ones.
N = 200
df = pd.DataFrame()
df["start"] = np.random.uniform(0, 20, size=N)
df["stop"] = df["start"] + np.random.choice([np.random.uniform(0, 0.1),
np.random.uniform(1., 2.)], p=[0.98, 0.02], size=N)
df["dist"] = np.random.random(size=N)
... and different styles
inferno_r color map, rounded or butted linecaps, semitransparent, bolder edges, etc (click for full size)
Here's a start. Let me know if this is what you had in mind.
I began with your data sample and put trip_start and trip_stop into POSIXct format before continuing with the code below.
library(tidyverse)
library(lubridate)
dat = dat %>%
mutate(start=(hour(trip_start)*60 + minute(trip_start) + second(trip_start))/(24*60) + wday(trip_start),
stop=(hour(trip_stop)*60 + minute(trip_stop) + second(trip_stop))/(24*60) + wday(trip_stop),
tod = case_when(hour(trip_start) < 6 ~ "night",
hour(trip_start) < 12 ~ "morning",
hour(trip_start) < 18 ~ "afternoon",
hour(trip_start) < 24 ~ "evening"))
ggplot(dat) +
geom_segment(aes(x=start, xend=stop,
y=trip_start,
yend=trip_stop,
colour=tod),
size=5, show.legend = FALSE) +
coord_polar() +
scale_y_datetime(breaks=seq(as.POSIXct("2017-09-01"), as.POSIXct("2017-12-31"), by="week")) +
scale_x_continuous(limits=c(1,8), breaks=1:7,
labels=weekdays(x=as.Date(seq(7)+2, origin="1970-01-01"),
abbreviate=TRUE))+
expand_limits(y=as.POSIXct("2017-08-25")) +
theme_bw() +
scale_colour_manual(values=c(night="black", morning="orange",
afternoon="orange", evening="blue")) +
labs(x="",y="")
This could be achieved relatively straightforwardly with d3. I'll use your data to create a rough template of one basic possible approach. Here's what the result of this approach might look like:
The key ingredient is d3's radial line component that lets us define a line by plotting angle and radius (here's a recent answer showing another spiral graph, that answer started me down the path on this answer).
All we need to do is scale angle and radius to be able to use this effectively (for which we need the first time and last time in the dataset):
var angle = d3.scaleTime()
.domain([start,end])
.range([0,Math.PI * 2 * numberWeeks])
var radius = d3.scaleTime()
.domain([start,end])
.range([minInnerRadius,maxOuterRadius])
And from there we can create a spiral quite easily, we sample some dates throughout the interval and then pass them to the radial line function:
var spiral = d3.radialLine()
.curve(d3.curveCardinal)
.angle(angle)
.radius(radius);
Here's a quick demonstration of just the spiral covering your time period. I'm assuming a base familiarity with d3 for this answer, so have not touched on a few parts of the code.
Once we have that, it's just a matter of adding sections from the data. The most simple way would be to plainly draw a stroke with some width and color it appropriately. This requires the same as above, but rather than sampling points from the start and end times of the dataset, we just need the start and end times of each datum:
// append segments on spiral:
var segments = g.selectAll()
.data(data)
.enter()
.append("path")
.attr("d", function(d) {
return /* sample points and feed to spiral function here */;
})
.style("stroke-width", /* appropriate width here */ )
.style("stroke",function(d) { return /* color logic here */ })
This might look something like this (with data mouseover).
This is just a proof of concept, if you were looking for more control and a nicer look, you could create a polygonal path for each data entry and use both fill & stroke. As is, you'll have to make do with layering strokes to get borders if desired and svg manipulations like line capping options.
Also, as it's d3, and longer timespans may be hard to show all at once, you could show less time but rotate the spiral so that it animates through your time span, dropping off events at the end and creating them in the origin. The actual chart might need to be canvas for this to happen smoothly depending on number of nodes, but to convert to canvas is relatively trivial in this case.
For the sake of filling out the visualization a little with a legend and day labels, this is what I have.
Related
I have a dataframe with ID, START and END time stamp and another reference table with ID, TIME and WEIGHT columns. Now, I am trying to assign the weights to the df1 based on times.
If the time of df2 is in between start and end of df1 the corresponding weight should be assigned for the record in df1. I can simply use a left join but the problem is there might be two or 3 weights assigned for the same ID
df1:
ID START END
2591642409 2018-08-20 06:00:00 2018-08-20 16:59:59
2591642409 2018-08-20 17:00:00 2018-08-21 01:59:59
2591642409 2018-08-21 02:00:00 2018-08-21 14:59:59
2591642409 2018-08-21 15:00:00 2018-08-21 15:59:59
2591642409 2018-08-21 15:00:00 2018-08-21 15:59:59
2591642409 2018-08-21 15:00:00 2018-08-21 14:59:59
2591642409 2018-08-21 15:00:00 2018-08-21 14:59:59
2591642409 2018-08-21 16:00:00 2018-08-25 11:59:59
2626784515 2018-09-12 12:41:00 2018-09-12 17:59:59
2626784515 2018-09-12 18:00:00 2018-09-12 22:27:59
2626784515 2018-09-12 22:28:00 2018-09-13 23:32:59
2626784515 2018-09-14 00:00:00 2018-09-13 23:59:59
2631776057 2018-09-16 03:29:00 2018-09-16 12:39:59
2631776057 2018-09-16 12:40:00 2018-09-16 13:33:59
2631776057 2018-09-16 13:34:00 2018-09-16 14:10:59
2694817807 2018-10-31 10:30:00 2018-11-01 15:57:59
2694817807 2018-11-01 15:58:00 2018-11-02 22:59:59
2694817807 2018-11-02 23:00:00 2018-11-02 23:55:59
2694817807 2018-11-02 23:56:00 2018-11-09 00:18:59
2694817807 2018-11-09 00:19:00 2018-11-09 05:55:59
2694817807 2018-11-09 05:56:00 2018-11-09 08:34:59
2694817807 2018-11-09 08:35:00 2018-11-09 16:59:59
2694817807 2018-11-09 17:00:00 2018-11-10 04:29:59
2694817807 2018-11-10 04:30:00 2018-11-10 09:23:59
2694817807 2018-11-10 09:24:00 2018-11-11 03:09:59
2694817807 2018-11-11 03:10:00 2018-11-11 16:54:59
2694817807 2018-11-11 16:55:00 2018-11-11 20:55:59
2694817807 2018-11-11 20:56:00 2018-11-12 19:59:59
2711413129 2018-11-12 20:00:00 2018-11-13 04:20:59
df2:
ID TIME WEIGHT
2591642409 2018-08-15 01:42:13 3.38
2626784515 2018-09-12 14:56:03 3.7
2631776057 2018-09-16 07:05:45 3.7
2694817807 2018-10-31 14:21:54 4.5
2694817807 2018-11-09 05:29:52 4.8
2711413129 2018-11-12 17:14:26 4.8
Expected df:
ID START END WEIGHT
2591642409 2018-08-20 06:00:00 2018-08-20 16:59:59 3.38
2591642409 2018-08-20 17:00:00 2018-08-21 01:59:59 3.38
2591642409 2018-08-21 02:00:00 2018-08-21 14:59:59 3.38
2591642409 2018-08-21 15:00:00 2018-08-21 15:59:59 3.38
2591642409 2018-08-21 15:00:00 2018-08-21 15:59:59 3.38
2591642409 2018-08-21 15:00:00 2018-08-21 14:59:59 3.38
2591642409 2018-08-21 15:00:00 2018-08-21 14:59:59 3.38
2591642409 2018-08-21 16:00:00 2018-08-25 11:59:59 3.38
2626784515 2018-09-12 12:41:00 2018-09-12 17:59:59 3.7
2626784515 2018-09-12 18:00:00 2018-09-12 22:27:59 3.7
2626784515 2018-09-12 22:28:00 2018-09-13 23:32:59 3.7
2626784515 2018-09-14 00:00:00 2018-09-13 23:59:59 3.7
2631776057 2018-09-16 03:29:00 2018-09-16 12:39:59 3.7
2631776057 2018-09-16 12:40:00 2018-09-16 13:33:59 3.7
2631776057 2018-09-16 13:34:00 2018-09-16 14:10:59 3.7
2694817807 2018-10-31 10:30:00 2018-11-01 15:57:59 4.5
2694817807 2018-11-01 15:58:00 2018-11-02 22:59:59 4.5
2694817807 2018-11-02 23:00:00 2018-11-02 23:55:59 4.5
2694817807 2018-11-02 23:56:00 2018-11-09 00:18:59 4.5
2694817807 2018-11-09 00:19:00 2018-11-09 05:55:59 4.5
2694817807 2018-11-09 05:56:00 2018-11-09 08:34:59 4.8
2694817807 2018-11-09 08:35:00 2018-11-09 16:59:59 4.8
2694817807 2018-11-09 17:00:00 2018-11-10 04:29:59 4.8
2694817807 2018-11-10 04:30:00 2018-11-10 09:23:59 4.8
2694817807 2018-11-10 09:24:00 2018-11-11 03:09:59 4.8
2694817807 2018-11-11 03:10:00 2018-11-11 16:54:59 4.8
2694817807 2018-11-11 16:55:00 2018-11-11 20:55:59 4.8
2694817807 2018-11-11 20:56:00 2018-11-12 19:59:59 4.8
2711413129 2018-11-12 20:00:00 2018-11-13 04:20:59 4.8
I am using the following code
mask = (df2['TIME'] > df1['START']) & (df2['TIME'] < df1['END'])
df1['WEIGHTS'] = np.where(mask, df2['WEIGHTS'], '')
but it throws a value error saying
ValueError: Can only compare identically-labeled Series objects
I'd really appreciate if I can get some help.
You can't compare two series with different name from different dataframe in pandas. You have to either change the name or join the two dataframe. In this case, I believe join is the best choice. Since len(df1) != len(df2) After join the dataframe, you should able to use the code.
You could try first joining or merging the two frames first... then apply your filter
df1.set_index('ID', inplace=True)
df2.set_index('ID', inplace=True)
df = df1.join(df2)
df_filtered = df[(df['TIME'] > df['START']) & (df['TIME'] < df['END'])]
I'm still a newbie to matplotlib. Currently, I have below dataset for plotting:
Date Open High Low Close
Trade_Date
2018-01-02 736696.0 42.45 42.45 41.45 41.45
2018-01-03 736697.0 41.60 41.70 40.70 40.95
2018-01-04 736698.0 40.90 41.05 40.20 40.25
2018-01-05 736699.0 40.35 41.60 40.35 41.50
2018-01-08 736702.0 40.20 40.20 37.95 38.00
2018-01-09 736703.0 37.15 39.00 37.15 38.00
2018-01-10 736704.0 38.70 38.70 37.15 37.25
2018-01-11 736705.0 37.50 37.50 36.55 36.70
2018-01-12 736706.0 37.00 37.40 36.90 37.20
2018-01-15 736709.0 37.50 37.70 37.15 37.70
2018-01-16 736710.0 37.80 38.25 37.45 37.95
2018-01-17 736711.0 38.00 38.05 37.65 37.75
2018-01-18 736712.0 38.00 38.20 37.70 37.75
2018-01-19 736713.0 36.70 37.10 35.30 36.45
2018-01-22 736716.0 36.25 36.25 35.50 36.10
2018-01-23 736717.0 36.20 36.30 35.65 36.00
2018-01-24 736718.0 35.80 36.00 35.60 36.00
2018-01-25 736719.0 36.10 36.10 35.45 35.45
2018-01-26 736720.0 35.50 35.75 35.00 35.00
2018-01-29 736723.0 34.80 35.00 33.65 33.70
2018-01-30 736724.0 33.70 34.45 33.65 33.90
I've converted the date value to number using mdates.date2num
After that, I've tried to plot candlestick graph with codes below:
f1, ax = plt.subplots(figsize= (10,5))
candlestick_ohlc(ax, ohlc.values, width=.6, colorup='red', colordown='green')
ax.xaxis.set_major_formatter(mdates.DateFormatter('%Y-%m-%d'))
plt.show()
However, I'm still getting the graph with gaps.
I've tried the possible solution from How do I plot only weekdays using Python's matplotlib candlestick?
However, I was not able to solve my problem with the solution above.
Can anyone kindly help me with this issue?
Thanks!
I am trying to get the last 8 hrs of data from a dataframe. Is there any way to achieve this?
Suppose I have a dataframe with timestamp:
Time X Y
19/8/2017 10:00:00 824.93 6.42
19/8/2017 10:20:00 825.88 6.41
19/8/2017 10:40:00 825.59 6.45
19/8/2017 11:00:00 826.18 6.6
19/8/2017 11:20:00 825.2 6.62
19/8/2017 11:40:00 825.3 6.65
19/8/2017 12:00:00 824.72 6.54
19/8/2017 12:20:00 825.72 6.45
19/8/2017 12:40:00 824.99 6.35
19/8/2017 13:00:00 824.63 6.31
19/8/2017 13:20:00 825.87 6.34
19/8/2017 13:40:00 826.12 6.29
19/8/2017 14:00:00 825.41 6.31
19/8/2017 14:20:00 825.34 6.36
19/8/2017 14:40:00 825.15 6.39
19/8/2017 15:00:00 824.87 6.15
19/8/2017 15:20:00 825.57 6.17
19/8/2017 15:40:00 825.35 6.2
19/8/2017 16:00:00 824.94 6.26
19/8/2017 16:20:00 825.86 6.39
19/8/2017 16:40:00 824.74 6.39
19/8/2017 17:00:00 823.46 6.33
19/8/2017 17:20:00 824.46 6.41
19/8/2017 17:40:00 824.67 6.4
19/8/2017 18:00:00 824.38 6.36
19/8/2017 18:20:00 825.23 6.37
19/8/2017 18:40:00 826.09 6.37
19/8/2017 19:00:00 824.77 6.32
19/8/2017 19:20:00 824.18 6.34
19/8/2017 19:40:00 823.82 6.33
19/8/2017 20:00:00 825 6.39
19/8/2017 20:20:00 825.01 6.4
19/8/2017 20:40:00 824.24 6.45
19/8/2017 21:00:00 817.65 6.17
19/8/2017 21:20:00 781.58 4.92
19/8/2017 21:40:00 739.89 3.51
I want to extract last 8 hours of data from this dataframe.
I tried this but nothing i can do:
last8 = (df["Time"] == (dfs["Time"].shift() + Timedelta(hours = 8))).cumsum()
Any help would be appreciated.
To get last 8 hours from the max time (last time in this case)
In [2178]: df[df['Time'] > df['Time'].max() - pd.Timedelta(hours=8)]
Out[2178]:
Time X Y
12 2017-08-19 14:00:00 825.41 6.31
13 2017-08-19 14:20:00 825.34 6.36
14 2017-08-19 14:40:00 825.15 6.39
15 2017-08-19 15:00:00 824.87 6.15
16 2017-08-19 15:20:00 825.57 6.17
17 2017-08-19 15:40:00 825.35 6.20
18 2017-08-19 16:00:00 824.94 6.26
19 2017-08-19 16:20:00 825.86 6.39
20 2017-08-19 16:40:00 824.74 6.39
21 2017-08-19 17:00:00 823.46 6.33
22 2017-08-19 17:20:00 824.46 6.41
23 2017-08-19 17:40:00 824.67 6.40
24 2017-08-19 18:00:00 824.38 6.36
25 2017-08-19 18:20:00 825.23 6.37
26 2017-08-19 18:40:00 826.09 6.37
27 2017-08-19 19:00:00 824.77 6.32
28 2017-08-19 19:20:00 824.18 6.34
29 2017-08-19 19:40:00 823.82 6.33
30 2017-08-19 20:00:00 825.00 6.39
31 2017-08-19 20:20:00 825.01 6.40
32 2017-08-19 20:40:00 824.24 6.45
33 2017-08-19 21:00:00 817.65 6.17
34 2017-08-19 21:20:00 781.58 4.92
35 2017-08-19 21:40:00 739.89 3.51
Try this:
df[df.Time >= pd.datetime.now() - pd.Timedelta('8H')]
I want to code a screener for U.S. stocks listed in AMEX, NYSE,NASDAQ.But I have difficulty in dealing with monthly stock price information.
I have made some code below to get monthly stock price from Yahoo!Finance.
Problem 1:How to get all ticker listed in AMEX, NYSE, NASDAQ?
Problem 2: I want do some conditional judgement like close_price >open_price for recent three months,and close_price of current month >close_price of previous month,close_price of previous month >close price of the month before the previous for every ticker(a judgement for uprising trend of some ticker)
Problem 3: perhaps something wrong with my output?
Could you please help me out of such situation?Thanks.
import pandas_datareader.data as web
import pandas as pd
all_data = {}
for ticker in ['MTW', 'CYH', 'PAH', 'FCX']:
#try:
all_data[ticker] = web.get_data_yahoo(ticker,'01/01/2016',interval='m')
print(all_data[ticker])
#print(all_data)
open_price = pd.DataFrame({tic: data['Open']
for tic, data in all_data.items()})
close_price = pd.DataFrame({tic: data['Close']
for tic, data in all_data.items()})
volume = pd.DataFrame({tic: data['Volume']
for tic, data in all_data.items()})
#date = pd.DataFrame({tic: data['Date']
#for tic, data in all_data.items()})
print(open_price,close_price,volume)
#except:
#print("Cant find",ticker)
output like this:
Open High Low Close Volume Adj Close
Date
2016-01-04 15.610001 16.07 11.73 15.740001 21098600 3.564441
2016-02-01 15.560000 16.07 13.17 15.850001 16000500 3.589351
2016-03-01 15.990001 17.40 4.00 4.330000 5162800 4.330000
2016-04-01 4.330000 5.87 4.21 5.700000 3370100 5.700000
2016-05-02 5.730000 6.15 5.38 5.700000 3662100 5.700000
2016-06-01 5.630000 5.89 5.03 5.450000 3600500 5.450000
2016-07-01 5.460000 5.79 4.97 5.570000 2903900 5.570000
2016-08-01 5.570000 5.84 4.62 4.900000 2983800 4.900000
2016-09-01 4.920000 4.96 4.27 4.790000 1460400 4.790000
2016-10-03 4.790000 4.86 3.98 4.040000 1311500 4.040000
2016-11-01 4.030000 6.04 3.65 5.960000 2759000 5.960000
2016-12-01 6.010000 6.30 5.70 5.980000 1880800 5.980000
2017-01-03 6.030000 7.57 5.65 7.110000 1886100 7.110000
MTW
Date
2016-01-04 15.610001
2016-02-01 15.560000
2016-03-01 15.990001
2016-04-01 4.330000
2016-05-02 5.730000
2016-06-01 5.630000
2016-07-01 5.460000
2016-08-01 5.570000
2016-09-01 4.920000
2016-10-03 4.790000
2016-11-01 4.030000
2016-12-01 6.010000
2017-01-03 6.030000 MTW
Date
2016-01-04 15.740001
2016-02-01 15.850001
2016-03-01 4.330000
2016-04-01 5.700000
2016-05-02 5.700000
2016-06-01 5.450000
2016-07-01 5.570000
2016-08-01 4.900000
2016-09-01 4.790000
2016-10-03 4.040000
2016-11-01 5.960000
2016-12-01 5.980000
2017-01-03 7.110000 MTW
Date
2016-01-04 21098600
2016-02-01 16000500
2016-03-01 5162800
2016-04-01 3370100
2016-05-02 3662100
2016-06-01 3600500
2016-07-01 2903900
2016-08-01 2983800
2016-09-01 1460400
2016-10-03 1311500
2016-11-01 2759000
2016-12-01 1880800
2017-01-03 1886100
Open High Low Close Volume Adj Close
Date
2016-01-04 25.610001 27.300001 17.650000 21.480000 5194900 17.752066
2016-02-01 21.300001 21.780001 12.860001 15.120001 6480900 12.495868
2016-03-01 15.180001 18.980000 14.910001 18.510000 4489500 15.297522
2016-04-01 18.440001 21.379999 18.000000 19.080000 3343100 19.080000
2016-05-02 16.150000 16.340000 11.810000 13.440000 4500200 13.440000
2016-06-01 13.330000 15.520000 11.700000 12.050000 4401000 12.050000
2016-07-01 12.080000 13.590000 11.430000 12.770000 3953800 12.770000
2016-08-01 12.900000 12.920000 9.660000 10.680000 5137800 10.680000
2016-09-01 10.700000 13.280000 9.810000 11.540000 5801900 11.540000
2016-10-03 11.610000 11.740000 4.880000 5.280000 8011100 5.280000
2016-11-01 5.290000 6.180000 4.150000 5.440000 6704700 5.440000
2016-12-01 5.420000 5.890000 5.240000 5.590000 3224200 5.590000
2017-01-03 5.690000 7.390000 5.410000 6.200000 3640200 6.200000
CYH MTW
Date
2016-01-04 25.610001 15.610001
2016-02-01 21.300001 15.560000
2016-03-01 15.180001 15.990001
2016-04-01 18.440001 4.330000
2016-05-02 16.150000 5.730000
2016-06-01 13.330000 5.630000
2016-07-01 12.080000 5.460000
2016-08-01 12.900000 5.570000
2016-09-01 10.700000 4.920000
2016-10-03 11.610000 4.790000
2016-11-01 5.290000 4.030000
2016-12-01 5.420000 6.010000
2017-01-03 5.690000 6.030000 CYH MTW
Date
2016-01-04 21.480000 15.740001
2016-02-01 15.120001 15.850001
2016-03-01 18.510000 4.330000
2016-04-01 19.080000 5.700000
2016-05-02 13.440000 5.700000
2016-06-01 12.050000 5.450000
2016-07-01 12.770000 5.570000
2016-08-01 10.680000 4.900000
2016-09-01 11.540000 4.790000
2016-10-03 5.280000 4.040000
2016-11-01 5.440000 5.960000
2016-12-01 5.590000 5.980000
2017-01-03 6.200000 7.110000 CYH MTW
Date
2016-01-04 5194900 21098600
2016-02-01 6480900 16000500
2016-03-01 4489500 5162800
2016-04-01 3343100 3370100
2016-05-02 4500200 3662100
2016-06-01 4401000 3600500
2016-07-01 3953800 2903900
2016-08-01 5137800 2983800
2016-09-01 5801900 1460400
2016-10-03 8011100 1311500
2016-11-01 6704700 2759000
2016-12-01 3224200 1880800
2017-01-03 3640200 1886100
Open High Low Close Volume Adj Close
Date
2016-01-04 12.35 12.36 6.48 7.63 4767700 7.63
2016-02-01 7.50 8.18 5.25 7.09 5294200 7.09
2016-03-01 7.01 9.48 6.21 8.60 3426300 8.60
2016-04-01 8.38 10.91 8.04 10.30 2084900 10.30
2016-05-02 10.24 10.55 8.39 9.49 2357400 9.49
2016-06-01 9.38 11.06 7.99 8.88 2065000 8.88
2016-07-01 8.79 9.39 7.77 9.20 1490500 9.20
2016-08-01 9.19 10.19 8.68 9.05 1692000 9.05
2016-09-01 9.07 9.75 7.82 8.11 3738300 8.11
2016-10-03 8.13 8.22 6.85 7.29 3499600 7.29
2016-11-01 7.38 9.47 7.16 9.35 2931900 9.35
2016-12-01 9.45 10.59 9.26 9.81 1761000 9.81
2017-01-03 9.90 12.11 9.64 12.00 2278800 12.00
CYH MTW PAH
Date
2016-01-04 25.610001 15.610001 12.35
2016-02-01 21.300001 15.560000 7.50
2016-03-01 15.180001 15.990001 7.01
2016-04-01 18.440001 4.330000 8.38
2016-05-02 16.150000 5.730000 10.24
2016-06-01 13.330000 5.630000 9.38
2016-07-01 12.080000 5.460000 8.79
2016-08-01 12.900000 5.570000 9.19
2016-09-01 10.700000 4.920000 9.07
2016-10-03 11.610000 4.790000 8.13
2016-11-01 5.290000 4.030000 7.38
2016-12-01 5.420000 6.010000 9.45
2017-01-03 5.690000 6.030000 9.90 CYH MTW PAH
Date
2016-01-04 21.480000 15.740001 7.63
2016-02-01 15.120001 15.850001 7.09
2016-03-01 18.510000 4.330000 8.60
2016-04-01 19.080000 5.700000 10.30
2016-05-02 13.440000 5.700000 9.49
2016-06-01 12.050000 5.450000 8.88
2016-07-01 12.770000 5.570000 9.20
2016-08-01 10.680000 4.900000 9.05
2016-09-01 11.540000 4.790000 8.11
2016-10-03 5.280000 4.040000 7.29
2016-11-01 5.440000 5.960000 9.35
2016-12-01 5.590000 5.980000 9.81
2017-01-03 6.200000 7.110000 12.00 CYH MTW PAH
Date
2016-01-04 5194900 21098600 4767700
2016-02-01 6480900 16000500 5294200
2016-03-01 4489500 5162800 3426300
2016-04-01 3343100 3370100 2084900
2016-05-02 4500200 3662100 2357400
2016-06-01 4401000 3600500 2065000
2016-07-01 3953800 2903900 1490500
2016-08-01 5137800 2983800 1692000
2016-09-01 5801900 1460400 3738300
2016-10-03 8011100 1311500 3499600
2016-11-01 6704700 2759000 2931900
2016-12-01 3224200 1880800 1761000
2017-01-03 3640200 1886100 2278800
Open High Low Close Volume Adj Close
Date
2016-01-04 6.60 6.760000 3.52 4.60 74784300 4.60
2016-02-01 4.50 7.970000 4.26 7.63 59604100 7.63
2016-03-01 7.84 11.450000 7.26 10.34 54212800 10.34
2016-04-01 9.92 14.060000 8.76 14.00 52896300 14.00
2016-05-02 14.03 14.040000 10.22 11.08 46463800 11.08
2016-06-01 11.08 11.930000 9.82 11.14 39599200 11.14
2016-07-01 11.04 13.590000 10.27 12.96 37208000 12.96
2016-08-01 13.01 13.040000 10.18 10.29 26767400 10.29
2016-09-01 10.32 11.140000 9.43 10.86 32267800 10.86
2016-10-03 10.98 11.200000 9.24 11.18 29236000 11.18
2016-11-01 11.31 16.420000 10.50 15.35 43726700 15.35
2016-12-01 15.50 16.080000 13.13 13.19 27355600 13.19
2017-01-03 13.49 17.059999 13.41 16.25 32808900 16.25
CYH FCX MTW PAH
Date
2016-01-04 25.610001 6.60 15.610001 12.35
2016-02-01 21.300001 4.50 15.560000 7.50
2016-03-01 15.180001 7.84 15.990001 7.01
2016-04-01 18.440001 9.92 4.330000 8.38
2016-05-02 16.150000 14.03 5.730000 10.24
2016-06-01 13.330000 11.08 5.630000 9.38
2016-07-01 12.080000 11.04 5.460000 8.79
2016-08-01 12.900000 13.01 5.570000 9.19
2016-09-01 10.700000 10.32 4.920000 9.07
2016-10-03 11.610000 10.98 4.790000 8.13
2016-11-01 5.290000 11.31 4.030000 7.38
2016-12-01 5.420000 15.50 6.010000 9.45
2017-01-03 5.690000 13.49 6.030000 9.90 CYH FCX MTW PAH
Date
2016-01-04 21.480000 4.60 15.740001 7.63
2016-02-01 15.120001 7.63 15.850001 7.09
2016-03-01 18.510000 10.34 4.330000 8.60
2016-04-01 19.080000 14.00 5.700000 10.30
2016-05-02 13.440000 11.08 5.700000 9.49
2016-06-01 12.050000 11.14 5.450000 8.88
2016-07-01 12.770000 12.96 5.570000 9.20
2016-08-01 10.680000 10.29 4.900000 9.05
2016-09-01 11.540000 10.86 4.790000 8.11
2016-10-03 5.280000 11.18 4.040000 7.29
2016-11-01 5.440000 15.35 5.960000 9.35
2016-12-01 5.590000 13.19 5.980000 9.81
2017-01-03 6.200000 16.25 7.110000 12.00 CYH FCX MTW PAH
Date
2016-01-04 5194900 74784300 21098600 4767700
2016-02-01 6480900 59604100 16000500 5294200
2016-03-01 4489500 54212800 5162800 3426300
2016-04-01 3343100 52896300 3370100 2084900
2016-05-02 4500200 46463800 3662100 2357400
2016-06-01 4401000 39599200 3600500 2065000
2016-07-01 3953800 37208000 2903900 1490500
2016-08-01 5137800 26767400 2983800 1692000
2016-09-01 5801900 32267800 1460400 3738300
2016-10-03 8011100 29236000 1311500 3499600
2016-11-01 6704700 43726700 2759000 2931900
2016-12-01 3224200 27355600 1880800 1761000
2017-01-03 3640200 32808900 1886100 2278800
I am learning how to filter dates on a Pandas data frame and need some help with the following please. This is my original data frame (from this data):
data
Out[120]:
Open High Low Last Volume NumberOfTrades BidVolume AskVolume
Timestamp
2014-03-04 09:30:00 1783.50 1784.50 1783.50 1784.50 171 17 29 142
2014-03-04 09:31:00 1784.75 1785.75 1784.50 1785.25 28 21 10 18
2014-03-04 09:32:00 1785.00 1786.50 1785.00 1786.50 81 19 4 77
2014-03-04 09:33:00 1786.00 1786.00 1785.25 1785.25 41 14 8 33
2014-03-04 09:34:00 1785.00 1785.25 1784.75 1785.25 11 8 2 9
2014-03-04 09:35:00 1785.50 1786.75 1785.50 1785.75 49 27 13 36
2014-03-04 09:36:00 1786.00 1786.00 1785.25 1785.75 12 8 3 9
2014-03-04 09:37:00 1786.00 1786.25 1785.25 1785.25 15 8 10 5
2014-03-04 09:38:00 1785.50 1785.50 1784.75 1785.25 24 17 17 7
data.dtypes
Out[118]:
Open float64
High float64
Low float64
Last float64
Volume int64
NumberOfTrades int64
BidVolume int64
AskVolume int64
dtype: object
I then resampled to 5 minute sections:
five_min = data.resample('5T').sum()
And look for the high volume days:
max_volume = five_min.Volume.at_time('9:30') > 65000
I then try to get the days high volume days as follows:
five_min.Volume = max_volume[max_volume == True]
for_high_vol = five_min.Volume.dropna()
for_high_vol
Timestamp
2014-03-21 09:30:00 True
2014-04-11 09:30:00 True
2014-04-16 09:30:00 True
2014-04-17 09:30:00 True
2014-07-18 09:30:00 True
2014-07-31 09:30:00 True
2014-09-19 09:30:00 True
2014-10-07 09:30:00 True
2014-10-10 09:30:00 True
2014-10-14 09:30:00 True
2014-10-15 09:30:00 True
2014-10-16 09:30:00 True
2014-10-17 09:30:00 True
I would like to use the index from "for_high_vol" to select all of the days from the original "data" Pandas dataframe.
Im sure there are much better was to approach this so can someone please show me the simplest way to do this?
IIUC, you can do it this way:
x.ix[(x.groupby(pd.Grouper(key='Timestamp', freq='5T'))['Volume'].transform('sum') > 65000)
&
(x.Timestamp.dt.hour==9)
&
(x.Timestamp.dt.minute>=30) & (x.Timestamp.dt.minute<=34)]
in order to set index back:
x.ix[(x.groupby(pd.Grouper(key='Timestamp', freq='5T'))['Volume'].transform('sum') > 65000)
&
(x.Timestamp.dt.hour==9)
&
(x.Timestamp.dt.minute>=30) & (x.Timestamp.dt.minute<=34)].set_index('Timestamp')
PS Timestamp is a regular column in my DF, not an index
Explanation:
resample / group our DF by 5 minutes interval, calculate the sum of Volume for each group and assign this sum to all rows in the group. For example in the example below 332 - is the sum of Volume in the first 5-min group
In [41]: (x.groupby(pd.Grouper(key='Timestamp', freq='5T'))['Volume'].transform('sum')).head(10)
Out[41]:
0 332
1 332
2 332
3 332
4 332
5 113
6 113
7 113
8 113
9 113
dtype: int64
filter time - the conditions are self-explanatory:
(x.Timestamp.dt.hour==9) & (x.Timestamp.dt.minute>=30) & (x.Timestamp.dt.minute<=34)].set_index('Timestamp')
and finally combine all conditions (filters) together - pass it to .ix[] indexer and set index back to Timestamp:
x.ix[(x.groupby(pd.Grouper(key='Timestamp', freq='5T'))['Volume'].transform('sum') > 65000)
&
(x.Timestamp.dt.hour==9)
&
(x.Timestamp.dt.minute>=30) & (x.Timestamp.dt.minute<=34)].set_index('Timestamp')
Output:
Out[32]:
Timestamp Open High Low Last Volume NumberOfTrades BidVolume AskVolume
5011 2014-03-21 09:30:00 1800.75 1802.50 1800.00 1802.25 30181 6006 13449 16732
5012 2014-03-21 09:31:00 1802.50 1803.25 1802.25 1802.50 15588 3947 5782 9806
5013 2014-03-21 09:32:00 1802.50 1803.75 1802.25 1803.25 16409 3994 6867 9542
5014 2014-03-21 09:33:00 1803.00 1803.50 1802.75 1803.25 10790 3158 4781 6009
5015 2014-03-21 09:34:00 1803.25 1804.75 1803.25 1804.75 13377 3466 4690 8687
11086 2014-04-11 09:30:00 1744.75 1744.75 1743.00 1743.50 21504 5876 11178 10326
11087 2014-04-11 09:31:00 1743.50 1746.50 1743.25 1746.00 21582 6191 8830 12752
11088 2014-04-11 09:32:00 1746.00 1746.50 1744.25 1745.75 18961 5214 9521 9440
11089 2014-04-11 09:33:00 1746.00 1746.25 1744.00 1744.25 12832 3658 7219 5613
11090 2014-04-11 09:34:00 1744.25 1744.25 1742.00 1742.75 15478 4919 8912 6566
12301 2014-04-16 09:30:00 1777.50 1778.25 1776.25 1777.00 21178 5431 10775 10403
12302 2014-04-16 09:31:00 1776.75 1779.25 1776.50 1778.50 16456 4400 6351 10105
12303 2014-04-16 09:32:00 1778.50 1779.25 1777.25 1777.50 9956 3015 5810 4146
12304 2014-04-16 09:33:00 1777.50 1778.00 1776.25 1776.25 8724 2470 5326 3398
12305 2014-04-16 09:34:00 1776.25 1777.00 1775.50 1776.25 9566 2968 5098 4468
12706 2014-04-17 09:30:00 1781.50 1782.50 1781.25 1782.25 16474 4583 7510 8964
12707 2014-04-17 09:31:00 1782.25 1782.50 1781.00 1781.25 10328 2587 6310 4018
12708 2014-04-17 09:32:00 1781.25 1782.25 1781.00 1781.25 9072 2142 4618 4454
12709 2014-04-17 09:33:00 1781.00 1781.75 1780.25 1781.25 17866 3807 10665 7201
12710 2014-04-17 09:34:00 1781.50 1782.25 1780.50 1781.75 11322 2523 5538 5784
38454 2014-07-18 09:30:00 1893.50 1893.75 1892.50 1893.00 24864 5135 13874 10990
38455 2014-07-18 09:31:00 1892.75 1893.50 1892.75 1892.75 8003 1751 3571 4432
38456 2014-07-18 09:32:00 1893.00 1893.50 1892.75 1893.50 7062 1680 3454 3608
38457 2014-07-18 09:33:00 1893.25 1894.25 1893.00 1894.25 10581 1955 3925 6656
38458 2014-07-18 09:34:00 1894.25 1895.25 1894.00 1895.25 15309 3347 5516 9793
42099 2014-07-31 09:30:00 1886.25 1886.25 1884.25 1884.75 21668 5857 11910 9758
42100 2014-07-31 09:31:00 1884.50 1884.75 1882.25 1883.00 17487 5186 11403 6084
42101 2014-07-31 09:32:00 1883.00 1884.50 1882.50 1884.00 13174 3782 4791 8383
42102 2014-07-31 09:33:00 1884.25 1884.50 1883.00 1883.25 9095 2814 5299 3796
42103 2014-07-31 09:34:00 1883.25 1884.25 1883.00 1884.25 7593 2528 3794 3799
... ... ... ... ... ... ... ... ... ...
193508 2016-01-21 09:30:00 1838.00 1838.75 1833.00 1834.00 22299 9699 12666 9633
193509 2016-01-21 09:31:00 1834.00 1836.50 1833.00 1834.50 8851 4520 4010 4841
193510 2016-01-21 09:32:00 1834.25 1835.25 1832.50 1833.25 7957 3672 3582 4375
193511 2016-01-21 09:33:00 1833.00 1838.50 1832.00 1838.00 12902 5564 5174 7728
193512 2016-01-21 09:34:00 1838.00 1841.50 1837.75 1840.50 13991 6130 6799 7192
199178 2016-02-10 09:30:00 1840.00 1841.75 1839.00 1840.75 13683 5080 6743 6940
199179 2016-02-10 09:31:00 1840.75 1842.00 1838.75 1841.50 11753 4623 5616 6137
199180 2016-02-10 09:32:00 1841.50 1844.75 1840.75 1843.00 16402 6818 8226 8176
199181 2016-02-10 09:33:00 1843.00 1843.50 1841.00 1842.00 14963 5402 8431 6532
199182 2016-02-10 09:34:00 1842.25 1843.50 1840.00 1840.00 8397 3475 4537 3860
200603 2016-02-16 09:30:00 1864.00 1866.25 1863.50 1864.75 19585 6865 9548 10037
200604 2016-02-16 09:31:00 1865.00 1865.50 1863.75 1864.25 16604 5936 8095 8509
200605 2016-02-16 09:32:00 1864.25 1864.75 1862.75 1863.50 10126 4713 5591 4535
200606 2016-02-16 09:33:00 1863.25 1863.75 1861.50 1862.25 9648 3786 5824 3824
200607 2016-02-16 09:34:00 1862.25 1863.50 1861.75 1862.25 10748 4143 5413 5335
205058 2016-03-02 09:30:00 1952.75 1954.25 1952.00 1952.75 19812 6684 10350 9462
205059 2016-03-02 09:31:00 1952.75 1954.50 1952.25 1953.50 10163 4236 3884 6279
205060 2016-03-02 09:32:00 1953.50 1954.75 1952.25 1952.50 15771 5519 8135 7636
205061 2016-03-02 09:33:00 1952.75 1954.50 1952.50 1953.75 9556 3583 3768 5788
205062 2016-03-02 09:34:00 1953.75 1954.75 1952.25 1952.50 11898 4463 6459 5439
209918 2016-03-18 09:30:00 2027.50 2028.25 2026.50 2028.00 38092 8644 17434 20658
209919 2016-03-18 09:31:00 2028.00 2028.25 2026.75 2027.25 11631 3209 6384 5247
209920 2016-03-18 09:32:00 2027.25 2027.75 2027.00 2027.50 9664 3270 5080 4584
209921 2016-03-18 09:33:00 2027.50 2027.75 2026.75 2026.75 10610 3117 5358 5252
209922 2016-03-18 09:34:00 2026.75 2027.00 2026.00 2026.50 8076 3022 4670 3406
227722 2016-05-20 09:30:00 2034.25 2035.25 2033.50 2034.50 30272 7815 16098 14174
227723 2016-05-20 09:31:00 2034.75 2035.75 2034.50 2035.50 12997 3690 6458 6539
227724 2016-05-20 09:32:00 2035.50 2037.50 2035.50 2037.25 12661 3864 5233 7428
227725 2016-05-20 09:33:00 2037.25 2037.75 2036.50 2037.00 9057 2524 5190 3867
227726 2016-05-20 09:34:00 2037.00 2037.50 2036.75 2037.00 5190 1620 2748 2442
[255 rows x 9 columns]