Develop Reference

Python Overpass API nodes

What exactly are those nodes in the line
"way(50.746,7.154,50.748,7.157) ["highway"];"
I tried to find documentation but still didn't understand.The following is a usage example
import overpy
api = overpy.Overpass()
# fetch all ways and nodes
result = api.query("""
way(50.746,7.154,50.748,7.157) ["highway"];
(._;>;);
out body;
""")
for way in result.ways:
print("Name: %s" % way.tags.get("name", "n/a"))
print(" Highway: %s" % way.tags.get("highway", "n/a"))
print(" Nodes:")
for node in way.nodes:
print(" Lat: %f, Lon: %f" % (node.lat, node.lon))

How to backtest portfolio compositions using backtrader?

I have a csv file / pandas dataframe which looks like this. It contains various portfolio compositions for a portfolio which is re-balanced everyday according to my own calculations.
date asset percentage
4-Jan-21 AAPL 12.00%
4-Jan-21 TSM 1.00%
4-Jan-21 IBM 31.00%
4-Jan-21 KO 15.00%
4-Jan-21 AMD 41.00%
5-Jan-21 DELL 23.00%
5-Jan-21 TSM 12.20%
5-Jan-21 IBM 15.24%
5-Jan-21 KO 1.50%
5-Jan-21 NKE 7.50%
5-Jan-21 TSLA 9.50%
5-Jan-21 CSCO 3.30%
5-Jan-21 JPM 27.76%
6-Jan-21 AMD 45%
6-Jan-21 BA 0.50%
6-Jan-21 ORCL 54.50%
7-Jan-21 AAPL 50.00%
7-Jan-21 KO 50.00%
...
I want to test a strategy with a 12 asset portfolio.
AAPL,TSM,IBM,KO,AMD,DELL,NKE,TSLA,CSCO,JPM,BA,ORCL
So let's say on 4Jan2021, the portfolio's composition would be 12% in apple, 1% in TSM.. etc. I want to be able to check the prices and know how many I should be holding.
The next day, 5Jan2021, the composition will change to 23% in Dell.. etc, if the stock isn't in this list means its 0% for that day.
I have been looking at backtrader as a backtesting platform, however, the code I have seen in the repo mostly shows how to do stuff with indicators, like SMA cross over, RSI...
My question is: Is it possible to create and test a portfolio based on these compositions I have so I can check the return of this strategy? It would check this frame, and know how many stocks in a ticker to buy or sell on that particular day.
So the universe of stocks I am buying or sell is AAPL,TSM,IBM,KO,AMD,DELL,NKE,TSLA,CSCO,JPM,BA,ORCL
So on 4-Jan-21 it might look like,
dictionary['4Jan2021'] = {'AAPL':0.12,
'TSM':0.01,
'IBM':0.31,
'KO':0.15,
'AMD':0.41,}
On 5-Jan-21 it will look like,
dictionary['5Jan2021'] = {'DELL':0.23,
'TSM':0.122,
'IBM':0.1524,
'KO':0.015,
'NKE':0.075,
'TSLA':0.095,
'CSCO':0.033,
'JPM':0.2776,}
If the ticker isnt there means its 0%.
The portfolio composition needs to change everyday.

The first thing you will want to do it load your targets with your datas. I like
personally to attach the target to the dataline as I add it to backtrader.
tickers = {"FB": 0.25, "MSFT": 0.4, "TSLA": 0.35}
for ticker, target in tickers.items():
data = bt.feeds.YahooFinanceData(
dataname=ticker,
timeframe=bt.TimeFrame.Days,
fromdate=datetime.datetime(2019, 1, 1),
todate=datetime.datetime(2020, 12, 31),
reverse=False,
)
data.target = target
cerebro.adddata(data, name=ticker)
In next you will want to go through each data, and determine the current allocation. If the current allocation is too far from the desired allocation (threshold) you trade all datas.
Notice there is a buffer variable. This will reduce the overall value of the account for calculating units to trade. This helps avoid margin.
You will use a dictionary to track this information.
def next(self):
track_trades = dict()
total_value = self.broker.get_value() * (1 - self.p.buffer)
for d in self.datas:
track_trades[d] = dict()
value = self.broker.get_value(datas=[d])
allocation = value / total_value
units_to_trade = (d.target - allocation) * total_value / d.close[0]
track_trades[d]["units"] = units_to_trade
# Can check to make sure there is enough distance away from ideal to trade.
track_trades[d]["threshold"] = abs(d.target - allocation) > self.p.threshold
Check all the thresholds to determine if trading. If any of datas need trading, then all need trading.
rebalance = False
for values in track_trades.values():
if values['threshold']:
rebalance = True
if not rebalance:
return
Finally, execute your trades. Always sell first to generate cash in the account and avoid margins.
# Sell shares first
for d, value in track_trades.items():
if value["units"] < 0:
self.sell(d, size=value["units"])
# Buy shares second
for d, value in track_trades.items():
if value["units"] > 0:
self.buy(d, size=value["units"])
Here is the all of the code for your reference.
import datetime
import backtrader as bt
class Strategy(bt.Strategy):
params = (
("buffer", 0.05),
("threshold", 0.025),
)
def log(self, txt, dt=None):
""" Logging function fot this strategy"""
dt = dt or self.data.datetime[0]
if isinstance(dt, float):
dt = bt.num2date(dt)
print("%s, %s" % (dt.date(), txt))
def print_signal(self):
self.log(
f"o {self.datas[0].open[0]:7.2f} "
f"h {self.datas[0].high[0]:7.2f} "
f"l {self.datas[0].low[0]:7.2f} "
f"c {self.datas[0].close[0]:7.2f} "
f"v {self.datas[0].volume[0]:7.0f} "
)
def notify_order(self, order):
""" Triggered upon changes to orders. """
# Suppress notification if it is just a submitted order.
if order.status == order.Submitted:
return
# Print out the date, security name, order number and status.
type = "Buy" if order.isbuy() else "Sell"
self.log(
f"{order.data._name:<6} Order: {order.ref:3d} "
f"Type: {type:<5}\tStatus"
f" {order.getstatusname():<8} \t"
f"Size: {order.created.size:9.4f} Price: {order.created.price:9.4f} "
f"Position: {self.getposition(order.data).size:5.2f}"
)
if order.status == order.Margin:
return
# Check if an order has been completed
if order.status in [order.Completed]:
self.log(
f"{order.data._name:<6} {('BUY' if order.isbuy() else 'SELL'):<5} "
# f"EXECUTED for: {dn} "
f"Price: {order.executed.price:6.2f} "
f"Cost: {order.executed.value:6.2f} "
f"Comm: {order.executed.comm:4.2f} "
f"Size: {order.created.size:9.4f} "
)
def notify_trade(self, trade):
"""Provides notification of closed trades."""
if trade.isclosed:
self.log(
"{} Closed: PnL Gross {}, Net {},".format(
trade.data._name,
round(trade.pnl, 2),
round(trade.pnlcomm, 1),
)
)
def next(self):
track_trades = dict()
total_value = self.broker.get_value() * (1 - self.p.buffer)
for d in self.datas:
track_trades[d] = dict()
value = self.broker.get_value(datas=[d])
allocation = value / total_value
units_to_trade = (d.target - allocation) * total_value / d.close[0]
track_trades[d]["units"] = units_to_trade
# Can check to make sure there is enough distance away from ideal to trade.
track_trades[d]["threshold"] = abs(d.target - allocation) > self.p.threshold
rebalance = False
for values in track_trades.values():
if values['threshold']:
rebalance = True
if not rebalance:
return
# Sell shares first
for d, value in track_trades.items():
if value["units"] < 0:
self.sell(d, size=value["units"])
# Buy shares second
for d, value in track_trades.items():
if value["units"] > 0:
self.buy(d, size=value["units"])
if __name__ == "__main__":
cerebro = bt.Cerebro()
tickers = {"FB": 0.25, "MSFT": 0.4, "TSLA": 0.35}
for ticker, target in tickers.items():
data = bt.feeds.YahooFinanceData(
dataname=ticker,
timeframe=bt.TimeFrame.Days,
fromdate=datetime.datetime(2019, 1, 1),
todate=datetime.datetime(2020, 12, 31),
reverse=False,
)
data.target = target
cerebro.adddata(data, name=ticker)
cerebro.addstrategy(Strategy)
# Execute
cerebro.run()
####################################
############# EDIT ###############
####################################
There was an additional requiest for adding in variable allocations per day per security. The following code accomplishes that.
import datetime
import backtrader as bt
class Strategy(bt.Strategy):
params = (
("buffer", 0.05),
("threshold", 0.025),
)
def log(self, txt, dt=None):
""" Logging function fot this strategy"""
dt = dt or self.data.datetime[0]
if isinstance(dt, float):
dt = bt.num2date(dt)
print("%s, %s" % (dt.date(), txt))
def print_signal(self):
self.log(
f"o {self.datas[0].open[0]:7.2f} "
f"h {self.datas[0].high[0]:7.2f} "
f"l {self.datas[0].low[0]:7.2f} "
f"c {self.datas[0].close[0]:7.2f} "
f"v {self.datas[0].volume[0]:7.0f} "
)
def notify_order(self, order):
""" Triggered upon changes to orders. """
# Suppress notification if it is just a submitted order.
if order.status == order.Submitted:
return
# Print out the date, security name, order number and status.
type = "Buy" if order.isbuy() else "Sell"
self.log(
f"{order.data._name:<6} Order: {order.ref:3d} "
f"Type: {type:<5}\tStatus"
f" {order.getstatusname():<8} \t"
f"Size: {order.created.size:9.4f} Price: {order.created.price:9.4f} "
f"Position: {self.getposition(order.data).size:5.2f}"
)
if order.status == order.Margin:
return
# Check if an order has been completed
if order.status in [order.Completed]:
self.log(
f"{order.data._name:<6} {('BUY' if order.isbuy() else 'SELL'):<5} "
# f"EXECUTED for: {dn} "
f"Price: {order.executed.price:6.2f} "
f"Cost: {order.executed.value:6.2f} "
f"Comm: {order.executed.comm:4.2f} "
f"Size: {order.created.size:9.4f} "
)
def notify_trade(self, trade):
"""Provides notification of closed trades."""
if trade.isclosed:
self.log(
"{} Closed: PnL Gross {}, Net {},".format(
trade.data._name,
round(trade.pnl, 2),
round(trade.pnlcomm, 1),
)
)
def __init__(self):
for d in self.datas:
d.target = {
datetime.datetime.strptime(date, "%d-%b-%y").date(): allocation
for date, allocation in d.target.items()
}
def next(self):
date = self.data.datetime.date()
track_trades = dict()
total_value = self.broker.get_value() * (1 - self.p.buffer)
for d in self.datas:
if date not in d.target:
if self.getposition(d):
self.close(d)
continue
target_allocation = d.target[date]
track_trades[d] = dict()
value = self.broker.get_value(datas=[d])
current_allocation = value / total_value
net_allocation = target_allocation - current_allocation
units_to_trade = (
(net_allocation) * total_value / d.close[0]
)
track_trades[d]["units"] = units_to_trade
# Can check to make sure there is enough distance away from ideal to trade.
track_trades[d]["threshold"] = abs(net_allocation) > self.p.threshold
rebalance = False
for values in track_trades.values():
if values["threshold"]:
rebalance = True
if not rebalance:
return
# Sell shares first
for d, value in track_trades.items():
if value["units"] < 0:
self.sell(d, size=value["units"])
# Buy shares second
for d, value in track_trades.items():
if value["units"] > 0:
self.buy(d, size=value["units"])
if __name__ == "__main__":
cerebro = bt.Cerebro()
allocations = [
("AAPL", "4-Jan-21", 0.300),
("TSM", "4-Jan-21", 0.200),
("IBM", "4-Jan-21", 0.300),
("KO", "4-Jan-21", 0.2000),
("AMD", "4-Jan-21", 0.1000),
("DELL", "5-Jan-21", 0.200),
("TSM", "5-Jan-21", 0.20),
("IBM", "5-Jan-21", 0.1),
("KO", "5-Jan-21", 0.1),
("NKE", "5-Jan-21", 0.15),
("TSLA", "5-Jan-21", 0.10),
("CSCO", "5-Jan-21", 0.050),
("JPM", "5-Jan-21", 0.1),
("AMD", "6-Jan-21", 0.25),
("BA", "6-Jan-21", 0.25),
("ORCL", "6-Jan-21", 0.50),
("AAPL", "7-Jan-21", 0.5000),
("KO", "7-Jan-21", 0.5000),
]
ticker_names = list(set([alls[0] for alls in allocations]))
targets = {ticker: {} for ticker in ticker_names}
for all in allocations:
targets[all[0]].update({all[1]: all[2]})
for ticker, target in targets.items():
data = bt.feeds.YahooFinanceData(
dataname=ticker,
timeframe=bt.TimeFrame.Days,
fromdate=datetime.datetime(2020, 12, 21),
todate=datetime.datetime(2021, 1, 8),
reverse=False,
)
data.target = target
cerebro.adddata(data, name=ticker)
cerebro.addstrategy(Strategy)
cerebro.broker.setcash(1000000)
# Execute
cerebro.run()

How to loop function using a list of variables?

I have a function that prints OHLCV data for stock prices from a websocket. It works but I have to copy it for each variable (Var1 to Var14) to get each individual stock data. How would I automate this process given that I have list:
varlist = [var1, var2, var3...var14]
and my code is:
def process_messages_for_var1(msg):
if msg['e'] == 'error':
print(msg['m'])
# If message is a trade, print the OHLC data
else:
# Convert time into understandable structure
transactiontime = msg['k']['T'] / 1000
transactiontime = datetime.fromtimestamp(transactiontime).strftime('%d %b %Y %H:%M:%S')
# Process this message once websocket starts
print("{} - {} - Interval {} - Open: {} - Close: {} - High: {} - Low: {} - Volume: {}".
format(transactiontime,msg['s'],msg['k']['i'],msg['k']['o'],msg['k']['c'],msg['k']['h'],msg['k']['l'],msg['k']['v']))
# Also, put information into an array
kline_array_msg = "{},{},{},{},{},{}".format(
msg['k']['T'],msg['k']['o'],msg['k']['c'],msg['k']['h'],msg['k']['l'],msg['k']['v'])
# Insert at first position
kline_array_dct[var1].insert(0, kline_array_msg)
if (len(kline_array_dct[var1]) > window):
# Remove last message when res_array size is > of FIXED_SIZE
del kline_array_dct[var1][-1]
I'm hoping to get the following result (notice how function name also changes):
def process_messages_for_var2(msg):
if msg['e'] == 'error':
print(msg['m'])
# If message is a trade, print the OHLC data
else:
# Convert time into understandable structure
transactiontime = msg['k']['T'] / 1000
transactiontime = datetime.fromtimestamp(transactiontime).strftime('%d %b %Y %H:%M:%S')
# Process this message once websocket starts
print("{} - {} - Interval {} - Open: {} - Close: {} - High: {} - Low: {} - Volume: {}".
format(transactiontime,msg['s'],msg['k']['i'],msg['k']['o'],msg['k']['c'],msg['k']['h'],msg['k']['l'],msg['k']['v']))
# Also, put information into an array
kline_array_msg = "{},{},{},{},{},{}".format(
msg['k']['T'],msg['k']['o'],msg['k']['c'],msg['k']['h'],msg['k']['l'],msg['k']['v'])
# Insert at first position
kline_array_dct[var2].insert(0, kline_array_msg)
if (len(kline_array_dct[var2]) > window):
# Remove last message when res_array size is > of FIXED_SIZE
del kline_array_dct[var2][-1]

You can adjust the function so that it takes one of the vars as an argument. I.e.,
def process_messages(msg, var):
...
kline_array_dct[var].insert(0, kline_array_msg)
if (len(kline_array_dct[var]) > window):
# Remove last message when res_array size is > of FIXED_SIZE
del kline_array_dct[var][-1]

If the processes are generally the same, just define one of them, and give it more arguments:
def process_messages(msg, var)
Then, you can adjust your process code to run through each var when you call it. You can do this by removing the numbered vars in the process code:
if msg['e'] == 'error':
print(msg['m'])
# If message is a trade, print the OHLC data
else:
# Convert time into understandable structure
transactiontime = msg['k']['T'] / 1000
transactiontime = datetime.fromtimestamp(transactiontime).strftime('%d %b %Y %H:%M:%S')
# Process this message once websocket starts
print("{} - {} - Interval {} - Open: {} - Close: {} - High: {} - Low: {} - Volume: {}".
format(transactiontime,msg['s'],msg['k']['i'],msg['k']['o'],msg['k']['c'],msg['k']['h'],msg['k']['l'],msg['k']['v']))
# Also, put information into an array
kline_array_msg = "{},{},{},{},{},{}".format(
msg['k']['T'],msg['k']['o'],msg['k']['c'],msg['k']['h'],msg['k']['l'],msg['k']['v'])
# Insert at first position
kline_array_dct[var].insert(0, kline_array_msg)
if (len(kline_array_dct[var]) > window):
# Remove last message when res_array size is > of FIXED_SIZE
del kline_array_dct[var][-1]
Then, create a simple for loop to call the process for each var in the list:
for var in varList:
process_messages("msg", var)
The for loop will call the process for each var in the list.

Could not convert string to float: ...Sometimes

I'm having some strange things happen when converting strings to floats in a loop.
So when I use the following code exactly it writes:
[1468436874000, 0.00254071495719],
[1468528803000, 0.00341349353996],
[1468688596000, 0.000853373384991],
[1468871365000, 0.00256012015497],
It stops short, there should be about 30 lines more than that and those are the wrong calculations.
The function is:
def main():
minprice = pricelist('MIN')
maxprice = pricelist('MAX')
avgprice = pricelist('AVG')
avgsqft = sqftlist()
datelist = getdates()
index = fileparser()
with open('/home/andrew/Desktop/index3.htm', 'w') as file:
file.writelines(data[:index[0]])
for date, minprice, maxprice in zip(datelist, minprice, maxprice):
file.writelines('[%s, %s, %s],\n' % (date, minprice, maxprice))
file.writelines(data[index[1]:index[2]])
for date, avgprice in zip(datelist, avgprice):
file.writelines('[%s, %s],\n' % (date, avgprice))
file.writelines(data[index[3]:index[4]])
for date, avgprice, avgsqft in zip(datelist, avgprice, avgsqft):
file.writelines('[%s, %s],\n' % (date, ((float(avgprice))/(float(avgsqft)))))
file.writelines(data[index[5]:])
file.close()
The error is:
file.writelines('[%s, %s],\n' % (date, ((float(avgprice))/(float(avgsqft)))))
ValueError: could not convert string to float: .
Oddly, when I comment out the other for loops before it, the result is:
[1468436874000, 2.82644376127],
[1468528803000, 2.86702735915],
[1468688596000, 2.8546107764],
[1468871365000, 2.8546107764],
[1468871996000, 2.8546107764],
[1468919656000, 2.85383420662],
[1469004050000, 2.85189704903],
[1469116491000, 2.87361540168],
[1469189815000, 2.86059636119],
[1469276601000, 2.83694745621],
[1469367041000, 2.83903252711],
[1469547497000, 2.83848688853],
[1469649630000, 2.83803033196],
[1469736031000, 2.82327110329],
[1469790030000, 2.82650020338],
[1469876430000, 2.96552660866],
[1470022624000, 2.93407180385],
Moreover, when I use enumerate instead of zip (and make the appropriate changes), it works. I've examined both lists at the fifth item for anything unusual because that's where it's getting hung up, but there is nothing odd there in either list. Since it does work fine with enumerate I'll just do that for now. But I'm new to Python/programming in general and want to understand what exactly is causing this.
UPDATE Should have included this the first time.
# file.writelines(data[:index[0]+1])
# for date, minprice, maxprice in zip(datelist, minprice, maxprice):
# file.writelines('[%s, %s, %s],\n' % (date, minprice, maxprice))
# file.writelines(data[index[1]:index[2]+1])
# for date, avgprice in zip(datelist, avgprice):
# file.writelines('[%s, %s],\n' % (date, avgprice))
# file.writelines(data[index[3]:index[4]+1])
# time.sleep(1)
for date, avgprice, avgsqft in zip(datelist, avgprice, avgsqft):
# file.writelines(
print'[%s, %s],\n' % (date, ((float(avgprice))/(float(avgsqft))))
# file.writelines(data[index[5]:])
# file.close()
Prints... (correctly)
[1468436874000, 2.82644376127],
[1468528803000, 2.86702735915],
[1468688596000, 2.8546107764],
[1468871365000, 2.8546107764],
[1468871996000, 2.8546107764],
[1468919656000, 2.85383420662],
etc...

Debug by printing the values of avgprice and avgsqft in your code. You are getting some string as it's value which can not be converted to float

ADCPi- how to run without being root?

I have a Pi Model A, running the latest version of Raspbian. Plugged into it is an ADC-Pi (https://www.abelectronics.co.uk/products/3/Raspberry-Pi/17/ADC-Pi-V2---Raspberry-Pi-Analogue-to-Digital-converter) with various analog sensors.
Running the demo code (which is below)
If I use './adc_demo.py' it works fine
If I use 'sudo python3 adc_demo.py' I get the error 'Import error: No module named quick2wire.i2c'.
What can I do so I can run it using the latter statement? I have another script that runs a motor through the GPIO pins on the pi, and that needs to be ran as root- and I'm trying to merge the two scripts together.
adc_demo.py
#!/usr/bin/env python3
# read abelectronics ADC Pi board inputs
# uses quick2wire from http://quick2wire.com/
# See http://elinux.org/RPi_ADC_I2C_Python for full setup instructions
import quick2wire.i2c as i2c
import re
import time
adc_address1 = 0x68
adc_address2 = 0x69
adc_channel1 = 0x98
adc_channel2 = 0xB8
adc_channel3 = 0xD8
adc_channel4 = 0xF8
for line in open('/proc/cpuinfo').readlines():
m = re.match('(.*?)\s*:\s*(.*)', line)
if m:
(name, value) = (m.group(1), m.group(2))
if name == "Revision":
if value [-4:] in ('0002', '0003'):
i2c_bus = 0
else:
i2c_bus = 1
break
with i2c.I2CMaster(i2c_bus) as bus:
def getadcreading(address, channel):
bus.transaction(i2c.writing_bytes(address, channel))
time.sleep(0.05)
h, l, r = bus.transaction(i2c.reading(address,3))[0]
time.sleep(0.05)
h, l, r = bus.transaction(i2c.reading(address,3))[0]
t = (h << 8) | l
v = t * 0.000154
if v < 5.5:
return v
else: # must be a floating input
return 0.00
while True:
print("1: %f" % getadcreading(adc_address1, adc_channel1))
print("2: %f" % getadcreading(adc_address1, adc_channel2))
print("3: %f" % getadcreading(adc_address1, adc_channel3))
print("4: %f" % getadcreading(adc_address1, adc_channel4))
print("5: %f" % getadcreading(adc_address2, adc_channel1))
print("6: %f" % getadcreading(adc_address2, adc_channel2))
print("7: %f" % getadcreading(adc_address2, adc_channel3))
print("8: %f" % getadcreading(adc_address2, adc_channel4))
time.sleep(1)