Develop Reference

How do I add a daterange to open trades during when backtesting with backtrader?

I am trying to backtest a strategy where trades are only opened during 8.30 to 16.00 using backtrader.
Using the below attempt my code is running but returning no trades so my clsoing balane is the same as the opening. If I remove this filter my code is running correctly and trades are opening and closing so it is definitely the issue. Can anyone please help?
I have tried adding the datetime column of the data to a data feed using the below code:
` def __init__(self):
# Keep a reference to the "close" line in the data[0] dataseries
self.dataclose = self.datas[0].close
self.datatime = mdates.num2date(self.datas[0].datetime)
self.datatsi = self.datas[0].tsi
self.datapsar = self.datas[0].psar
self.databbmiddle = self.datas[0].bbmiddle
self.datastlower = self.datas[0].stlower
self.datastupper = self.datas[0].stupper
# To keep track of pending orders
self.order = None`
I then used the following code to try filter by this date range:
# Check if we are in the market
if not self.position:
current_time = self.datatime[0].time()
if datetime.time(8, 30) < current_time < datetime.time(16, 0):
if self.datatsi < 0 and self.datastupper[0] > self.dataclose[0] and self.datastlower[1] < self.dataclose[1] and self.dataclose[0] < self.databbmiddle[0] and self.datapsar[0] > self.dataclose[0]:
self.log('SELL CREATE, %.2f' % self.dataclose[0])
# Keep track of the created order to avoid a 2nd order
os = self.sell_bracket(size=100,price=sp1, stopprice=sp2, limitprice=sp3)
self.orefs = [o.ref for o in os]
else:
o1 = self.buy(exectype=bt.Order.Limit,price=bp1,transmit=False)
print('{}: Oref {} / Buy at {}'.format(self.datetime.date(), o1.ref, bp1))
o2 = self.sell(exectype=bt.Order.Stop,price=bp2,parent=o1,transmit=False)
print('{}: Oref {} / Sell Stop at {}'.format(self.datetime.date(), o2.ref, bp2))
o3 = self.sell(exectype=bt.Order.Limit,price=bp3,parent=o1,transmit=True)
print('{}: Oref {} / Sell Limit at {}'.format(self.datetime.date(), o3.ref, bp3))
self.orefs = [o1.ref, o2.ref, o3.ref] # self.sell(size=100, exectype=bt.Order.Limit, price=self.data.close[0]+16, parent=self.order, parent_bracket=bt.Order.Market)

TypeError: coercing to Unicode: need string or buffer, datetime.timedelta found

im currently getting this error
Traceback (most recent call last):
File "/Users/user/Documents/test.py", line 44, in <module>
get_slots(hours, appointments)
File "/Users/user/Documents/test.py", line 36, in get_slots
while start + duration <= end:
TypeError: coercing to Unicode: need string or buffer, datetime.timedelta found
My code:
from datetime import timedelta
import datetime
#notice the additional brackets to keep the 2 slots as two separate lists. So, 930-1230 is one slot, 1330-1400 is an another.
# HOURS AND APPOINTMENTS ARE GENERATED BY GATHERING DATA FROM DATABASE
hours = [[u'08:00', u'17:00']]
appointments = [(u'12:00', u'12:30'), (u'10:30', u'11:00')]
def get_slots(hours, appointments, duration=timedelta(hours=1)):
slots = sorted([(hours[0][0], hours[0][0])] + appointments + [(hours[0][1], hours[0][1])])
for start, end in ((slots[i][1], slots[i+1][0]) for i in range(len(slots)-1)):
assert start <= end, "Cannot attend all appointments"
while start + duration <= end:
json = []
json.append("{:%H:%M} - {:%H:%M}".format(start, start + duration))
start += duration
return json
if __name__ == "__main__":
get_slots(hours, appointments)
The code should output something like:
09:00 - 10:00
10:30 - 11:30
13:00 - 14:00
14:00 - 15:00
I found this code from Python - finding time slots

You have to convert both start and end string to datetime objects. See below example:
from datetime import timedelta
import datetime
#notice the additional brackets to keep the 2 slots as two separate lists. So, 930-1230 is one slot, 1330-1400 is an another.
# HOURS AND APPOINTMENTS ARE GENERATED BY GATHERING DATA FROM DATABASE
hours = [[u'08:00', u'17:00']]
appointments = [(u'12:00', u'12:30'), (u'10:30', u'11:00')]
def get_slots(hours, appointments, duration=timedelta(hours=1)):
slots = sorted([(hours[0][0], hours[0][0])] + appointments + [(hours[0][1], hours[0][1])])
for start, end in ((slots[i][1], slots[i+1][0]) for i in range(len(slots)-1)):
start = datetime.datetime.strptime(start, "%H:%M")
end = datetime.datetime.strptime(end, "%H:%M")
print(start+duration)
assert start <= end, "Cannot attend all appointments"
while start + duration <= end:
json = []
json.append("{:%H:%M} - {:%H:%M}".format(start, start + duration))
start += duration
return json
if __name__ == "__main__":
x = get_slots(hours, appointments)

How to loop function using a list of variables?

I have a function that prints OHLCV data for stock prices from a websocket. It works but I have to copy it for each variable (Var1 to Var14) to get each individual stock data. How would I automate this process given that I have list:
varlist = [var1, var2, var3...var14]
and my code is:
def process_messages_for_var1(msg):
if msg['e'] == 'error':
print(msg['m'])
# If message is a trade, print the OHLC data
else:
# Convert time into understandable structure
transactiontime = msg['k']['T'] / 1000
transactiontime = datetime.fromtimestamp(transactiontime).strftime('%d %b %Y %H:%M:%S')
# Process this message once websocket starts
print("{} - {} - Interval {} - Open: {} - Close: {} - High: {} - Low: {} - Volume: {}".
format(transactiontime,msg['s'],msg['k']['i'],msg['k']['o'],msg['k']['c'],msg['k']['h'],msg['k']['l'],msg['k']['v']))
# Also, put information into an array
kline_array_msg = "{},{},{},{},{},{}".format(
msg['k']['T'],msg['k']['o'],msg['k']['c'],msg['k']['h'],msg['k']['l'],msg['k']['v'])
# Insert at first position
kline_array_dct[var1].insert(0, kline_array_msg)
if (len(kline_array_dct[var1]) > window):
# Remove last message when res_array size is > of FIXED_SIZE
del kline_array_dct[var1][-1]
I'm hoping to get the following result (notice how function name also changes):
def process_messages_for_var2(msg):
if msg['e'] == 'error':
print(msg['m'])
# If message is a trade, print the OHLC data
else:
# Convert time into understandable structure
transactiontime = msg['k']['T'] / 1000
transactiontime = datetime.fromtimestamp(transactiontime).strftime('%d %b %Y %H:%M:%S')
# Process this message once websocket starts
print("{} - {} - Interval {} - Open: {} - Close: {} - High: {} - Low: {} - Volume: {}".
format(transactiontime,msg['s'],msg['k']['i'],msg['k']['o'],msg['k']['c'],msg['k']['h'],msg['k']['l'],msg['k']['v']))
# Also, put information into an array
kline_array_msg = "{},{},{},{},{},{}".format(
msg['k']['T'],msg['k']['o'],msg['k']['c'],msg['k']['h'],msg['k']['l'],msg['k']['v'])
# Insert at first position
kline_array_dct[var2].insert(0, kline_array_msg)
if (len(kline_array_dct[var2]) > window):
# Remove last message when res_array size is > of FIXED_SIZE
del kline_array_dct[var2][-1]

You can adjust the function so that it takes one of the vars as an argument. I.e.,
def process_messages(msg, var):
...
kline_array_dct[var].insert(0, kline_array_msg)
if (len(kline_array_dct[var]) > window):
# Remove last message when res_array size is > of FIXED_SIZE
del kline_array_dct[var][-1]

If the processes are generally the same, just define one of them, and give it more arguments:
def process_messages(msg, var)
Then, you can adjust your process code to run through each var when you call it. You can do this by removing the numbered vars in the process code:
if msg['e'] == 'error':
print(msg['m'])
# If message is a trade, print the OHLC data
else:
# Convert time into understandable structure
transactiontime = msg['k']['T'] / 1000
transactiontime = datetime.fromtimestamp(transactiontime).strftime('%d %b %Y %H:%M:%S')
# Process this message once websocket starts
print("{} - {} - Interval {} - Open: {} - Close: {} - High: {} - Low: {} - Volume: {}".
format(transactiontime,msg['s'],msg['k']['i'],msg['k']['o'],msg['k']['c'],msg['k']['h'],msg['k']['l'],msg['k']['v']))
# Also, put information into an array
kline_array_msg = "{},{},{},{},{},{}".format(
msg['k']['T'],msg['k']['o'],msg['k']['c'],msg['k']['h'],msg['k']['l'],msg['k']['v'])
# Insert at first position
kline_array_dct[var].insert(0, kline_array_msg)
if (len(kline_array_dct[var]) > window):
# Remove last message when res_array size is > of FIXED_SIZE
del kline_array_dct[var][-1]
Then, create a simple for loop to call the process for each var in the list:
for var in varList:
process_messages("msg", var)
The for loop will call the process for each var in the list.

Python: Finding time taken for each event in dataframe based on condition

I have a df with two columns, timestamp & eventType.
timestamp is ordered in chronological order, and eventType can be either ['start', 'change', 'end', resolve].
['start', 'change'] denotes the start of an event
['end','resolve'] denotes the end of an event
createdTime actionName
2020-03-16 18:28:14 start
2020-03-17 19:12:42 end
2020-03-18 19:56:10 change
2020-03-19 21:29:13 change
2020-03-20 21:42:06 end
2020-03-21 18:28:14 start
2020-03-21 19:12:42 resolve
2020-03-22 19:56:10 change
2020-03-22 21:29:13 change
2020-03-23 21:42:06 end
I wish to calculate the time delta between the each start/change event to the next end/resolve event.
An event can have several start/change statuses before it is
resolved, thus an event would need to take the initial start/change
status as the 1st start/change event time.
The output would need to be a list of time deltas taken for each event in the df
Thanks in advance :)
Edit
The expected outcome should be a list containing each time taken for each event.
event_times = ['24:44:28', '49:45.56', '0:44:28', '25:45:56']

Better late than never?
df['createdTime'] = pd.to_datetime(df.createdTime)
starts = ['start', 'change']
ends = ['end','resolve']
prev_status = 'end'
spans = []
for i in range(len(df)):
curr_status = df.actionName[i]
if curr_status in starts and prev_status in starts:
pass
elif curr_status in starts and prev_status in ends:
start_time = df.createdTime[i]
elif curr_status in ends and prev_status in starts:
t = df.createdTime[i] - start_time
hours = t.days * 24 + t.seconds // 3600
minutes = t.seconds % 3600 // 60
seconds = t.seconds % 60
spans.append(f"{hours}:{minutes}:{seconds}")
elif curr_status in ends and prev_status in ends:
raise ValueError (f"Two ends in a row at index {i}.")
else:
raise ValueError (f"Unrecognized action type at index {i}.")
prev_status = curr_status
print(spans)
gives
['24:44:28', '49:45:56', '0:44:28', '25:45:56']

Split up datetime interval according to labeled partition of a week

I have shift which is a datetime interval (a pair of datetimes). My weeks have a labeled partition (every week is the same: divided into parts, and each part has a label). I want to split up shift into labeled parts (i.e. into several subintervals), according to the partition of the week.
Example. Suppose shift is the interval 2019-10-21 18:30 - 2019-10-22 08:00, and the partition of the week is as follows: Monday to Friday 07:00 - 19:00 has label A, and the rest of the week has label B.
In this case the splitting of shift should be the following list of labeled subintervals:
2019-10-21 18:30 - 2019-10-21 19:00 with label A,
2019-10-21 19:00 - 2019-10-22 07:00 with label B, and
2019-10-22 07:00 - 2019-10-22 08:00 with label A.
How do I do this in general?
Input: a datetime interval (pair), and a labeled partition of the week (how to best represent this?)
Output: a list of labeled datetime intervals (pairs).
Note that shift can start in one week and end in another week (e.g. Sunday evening to Monday morning); each week does have the same labeled partition.

Here's a way to obtain the desired intervals:
from collections import namedtuple
from datetime import datetime, timedelta
import itertools as it
# Built-in as `it.pairwise` in Python 3.10+
def pairwise(iterable):
it = iter(iterable)
a = next(it, None)
for b in it:
yield (a, b)
a = b
def beginning_of_week(d: datetime) -> datetime:
''' Returns the datetime object for the beginning of the week the provided day is in. '''
return (d - timedelta(days=d.weekday())).replace(hour=0, minute=0, second=0, microsecond=0)
Partition = namedtuple('Partition', ('start', 'stop', 'label')) # output format
def _partition_shift_within_week(start: int, stop: int, partitions):
''' Splits the shift (defined by `start` and `stop`) into partitions within one week. '''
# Get partitions as ranges of absolute offsets from the beginning of the week in seconds
labels = (x for _, x in partitions)
absolute_offsets = it.accumulate(int(x.total_seconds()) for x, _ in partitions)
ranges = [range(x, y) for x, y in pairwise((0, *absolute_offsets))]
first_part_idx = [start in x for x in ranges].index(True)
last_part_idx = [stop in x for x in ranges].index(True)
for r, label in zip((ranges[i] for i in range(first_part_idx, last_part_idx + 1)), labels):
yield Partition(
timedelta(seconds=max(r.start, start)), # start of subinterval
timedelta(seconds=min(r.stop, stop)), # end of the subinterval
label
)
def _partition_shift_unjoined(shift, partitions):
''' Partitions a shift across weeks with partitions unjoined at the week edges. '''
start_monday = beginning_of_week(shift[0])
stop_monday = beginning_of_week(shift[1])
seconds_offsets = (
int((shift[0] - start_monday).total_seconds()),
*[604800] * ((stop_monday - start_monday).days // 7),
int((shift[1] - stop_monday).total_seconds()),
)
for x, y in pairwise(seconds_offsets):
num_weeks, x = divmod(x, 604800)
for part in _partition_shift_within_week(x, y - (y == 604800), partitions):
weeks_offset = timedelta(weeks=num_weeks)
yield Partition(
start_monday + weeks_offset + part.start,
start_monday + weeks_offset + part.stop,
part.label
)
def partition_shift(shift, partitions):
''' Partitions a shift across weeks. '''
results = []
for part in _partition_shift_unjoined(shift, partitions):
if len(results) and results[-1].label == part.label:
results[-1] = Partition(results[-1].start, part.stop, part.label)
else:
results.append(part)
return results
Usage example:
shift = (datetime(2019, 10, 21, 18, 30), datetime(2019, 10, 22, 8, 0))
# Partitions are stored as successive offsets from the beginning of the week
partitions = (
(timedelta(hours=7), 'B'), # Monday morning (midnight to 07:00)
(timedelta(hours=12), 'A'),
(timedelta(hours=12), 'B'), # Monday night & Tuesday morning (til 07:00)
(timedelta(hours=12), 'A'),
(timedelta(hours=12), 'B'), # Tuesday night & Wednesday morning (til 07:00)
(timedelta(hours=12), 'A'),
(timedelta(hours=12), 'B'), # Wednesday night & Thursday morning (til 07:00)
(timedelta(hours=12), 'A'),
(timedelta(hours=12), 'B'), # Thursday night & Friday morning (til 07:00)
(timedelta(hours=12), 'A'),
(timedelta(hours=53), 'B'), # Friday night & the weekend
)
for start, end, label in partition_shift(shift, partitions):
print(f"'{start}' - '{end}', label: {label}")
Output:
'2019-10-21 18:30:00' - '2019-10-21 19:00:00', label: A
'2019-10-21 19:00:00' - '2019-10-22 07:00:00', label: B
'2019-10-22 07:00:00' - '2019-10-22 08:00:00', label: A
This approach assumes that the partitions are input as successive offsets from the beginning of that week. The question did not specify how the partitions would be provided, so I choose to use this format. It's nice because it guarantees they do not overlap, and uses time deltas instead of being fixed to some particular date.
Converting other ways of specifying partitions into this one, or adapting this answer to work with other ways of specifying partitions has been left as an exercise to the reader.
Here's another usage example, using the same partitions as before, but a shift that starts in the previous week, thereby demonstrating that this approach works even when the shift spans multiple weeks.
shift = (datetime(2019, 10, 19, 18, 30), datetime(2019, 10, 22, 8, 0))
for start, end, label in partition_shift(shift, partitions):
print(f"'{start}' - '{end}', label: {label}")
Output:
'2019-10-19 18:30:00' - '2019-10-21 07:00:00', label: B
'2019-10-21 07:00:00' - '2019-10-21 19:00:00', label: A
'2019-10-21 19:00:00' - '2019-10-22 07:00:00', label: B
'2019-10-22 07:00:00' - '2019-10-22 08:00:00', label: A

I'd approach this by building the generic data structure, and then mapping week-minutes on top of it.
The generic structure looks like this:
class OrderedRangeMap:
""" ranges must be contiguous ; 0..limit """
def __init__(self, limit, default_value=""):
self.ranges = [(0,default_value),(limit,None)]
def find(self, key):
# could do bsearch
# what if value < self.ranges[0]?
kv = self.ranges[0]
if key < kv[0]:
return None,0,False
# what if value = self.ranges[0]?
# what if value == vl[0]?
for i,kv in enumerate(self.ranges):
k = kv[0]
if key < k:
return kvp,i-1,False
if key == k:
return kv,i,True
kvp = kv
# off the end
return None, len(self.ranges)-1, False
def add(self, skey, ekey, value):
newblock = (skey,value)
oldblock,si,sx = self.find(skey)
endblock,ei,ex = self.find(ekey)
if sx: #if start match, replace the oldblock
self.ranges[si] = newblock
else: #else insert after the oldblock
# bump
si += 1
ei += 1
self.ranges.insert(si,newblock)
if si == ei:
# insert the split block after that
self.ranges.insert(si+1,(ekey,oldblock[1]))
else:
# different blocks
# end block starts at new end point
self.ranges[ei] = (ekey,endblock[1])
# delete any in between
del self.ranges[si+1:ei]
# is that it?
def __getitem__(self, key):
block,index,match = self.find(key)
if index >= len(self.ranges) - 1:
return block[0], block[0], block[1]
return block[0], self.ranges[index+1][0], block[1]
def test_orm():
orm = OrderedRangeMap(100, "B")
assert orm.ranges == [(0,"B"),(100,None)]
# s/e in same block
orm.add(10,20, "A")
assert orm.ranges == [(0,"B"),(10,"A"),(20,"B"),(100,None)]
# s/e in same blocks, matches
orm.add(10,13, "a")
assert orm.ranges == [(0,"B"),(10,"a"),(13, "A"),(20,"B"),(100,None)]
# more blocks
orm.add(30,50, "c")
assert orm.ranges == [(0,"B"),(10,"a"),(13, "A"),(20,"B"),(30,"c"),(50,"B"),(100,None)]
# s/e in different blocks, no matches
orm.add(15,33, "d")
assert orm.ranges == [(0,"B"),(10,"a"),(13, "A"),(15,"d"),(33,"c"),(50,"B"),(100,None)]
# s/e in different blocks, s matches
orm.add(15,44, "e")
assert orm.ranges == [(0,"B"),(10,"a"),(13, "A"),(15,"e"),(44,"c"),(50,"B"),(100,None)]
# s/e in different blocks, s & e matches
orm.add(13,50, "f")
assert orm.ranges == [(0,"B"),(10,"a"),(13, "f"),(50,"B"),(100,None)]
# NOT tested: add outside of original range
test_orm()
(Edited to add:)
The upper layer converts from datetime to week minutes
import datetime
class WeekShiftLabels:
# this is assuming Monday=0
week_minutes = 7*24*60
def __init__(self, default_label="?"):
self.orm = OrderedRangeMap(self.week_minutes, default_label)
def add(self, dow, starttime, endtime, label):
dm = dow * 24*60
st = dm + t2m(starttime)
et = dm + t2m(endtime)
self.orm.add(st, et, label)
def __getitem__(self, dt):
wm = dt2wm(dt)
block,index,match = self.find(wm)
if index >= len(self.ranges) - 1:
return None
return block[1]
class WSLI:
# This doesn't handle modulo week_minutes
def __init__(self, wsl, sdt, edt):
self.wsl = wsl
self.base = sdt - datetime.timedelta(days=sdt.weekday())
t = sdt.time()
self.base -= datetime.timedelta(hours=t.hour,minutes=t.minute)
self.i = dt2wm(sdt)
self.em = dt2wm(edt)
def __next__(self):
if self.i < 0:
raise StopIteration
block,index,match = self.wsl.orm.find(self.i)
if not block:
raise StopIteration # or something else
start = wm2dt(self.base, self.i)
end = self.wsl.orm.ranges[index+1][0]
if end >= self.em:
end = self.em
self.i = -1
else:
self.i = end
end = wm2dt(self.base, end)
return start, end, block[1]
def __iter__(self):
return self
def __call__(self, sdt, edt):
return self.WSLI(self, sdt, edt)
def dt2wm(dt):
t = dt.time()
return dt.weekday() * 24*60 + t.hour*60 + t.minute
def wm2dt(base,wm):
return base + datetime.timedelta(minutes=wm)
def t2m(t):
return t.hour*60 + t.minute
def test_wsl():
wsl = WeekShiftLabels("B")
st = datetime.time(hour=7)
et = datetime.time(hour=19)
for dow in range(0,6):
wsl.add(dow, st, et, "A")
r = list(wsl(datetime.datetime(2019, 10, 21, 18, 30), datetime.datetime(2019, 10, 22, 8, 0)))
assert len(r) == 3
assert r[0]==(datetime.datetime(2019, 10, 21, 18, 30), datetime.datetime(2019, 10, 21, 19, 0), 'A')
assert r[1]==(datetime.datetime(2019, 10, 21, 19, 0), datetime.datetime(2019, 10, 22, 7, 0), 'B')
assert r[2]==(datetime.datetime(2019, 10, 22, 7, 0), datetime.datetime(2019, 10, 22, 8, 0), 'A')
test_wsl()

You have not defined what happens if one of your shift limits (start or end) lays inside one part of your week and the other limit lays outside. For example what happens if you are having
2019-10-21 18:30 - 2019-10-21 19:00
Is it A or B? You could either make rules that if one part is in "B" the label would be B or just test the start or end or you could take the average., etc. So I will show how to check if one specific datetime lays inside the intervals. I don't know of a library automating this task more than the datetime library.
datetime library
import datetime
now = datetime.datetime.now()
hour = now.hour
# day of the week as int, where Mon 0 and Sun 6
day = now.weekday()
if day < 5 and hour >= 7 and hour < 19:
label = "A"
else:
label = "B"
print(label)
You could also check if hour or day lays in a range or list. For example if you want to consider a break between 12 and 13 o'clock:
if hour in range(7, 13) or hour in range(13, 19):
# do something
docs for more info: https://docs.python.org/3.8/library/datetime.html
People also often recommend the Pendulum library but by scrolling over it's docs I can't see any method that makes your task easier than the above code. Of course you could do something like this (but it seems not to be easier for me; code is not tested):
alternate solution using pendulum
import pendulum
now = pendulum.now()
daystart = now.start_of('day')
weekstart = now.start_of('week')
if now < weekstart.add(days=5) and now > daystart.add(hours=7) and now < daystart.add(hours=19):
label = "A"
else:
label = "B"
pendulum docs: https://pendulum.eustace.io/docs
It must be stated here though that both solutions can be done in this way (or with some adjustments) in both libraries (pendulum and datetime) and probably in many others I haven't mentioned as well.
Bonus
Since you asked for a way to handle such things more generally I will show you one last thing how you could use the first solution and make it a bit more generic:
import datetime
gethour = lambda dt : dt.hour
getday = lambda dt : dt.weekday()
timeframes = {
"A": {
getday: range(0,6),
gethour: range(7,13) + range(13,19)
},
"break": {
getday: range(0,6),
gethour: [12]
}
}
default = "B"
now = datetime.datetime.now()
for tag, timeframe in timeframes.items():
label = tag
for getter, limit in timeframe.items():
if not getter(now) in limit:
label = default
break
if label != default:
break
print(label)