Develop Reference

Handling high frequency updates with streamlit

Summary
I want to use streamlit to create a dashboard of all the trades (buy and sell) happening in a given market. I connect to a websocket stream to receive data of BTCUSDT from the Binance exchange. Messages are received every ~0.1s and I would like to update my dashboard in ~0.09s.
How can you handle this kind of situation where messages are delivered at high frequency? With my code, I successfully create a dashboard but it doesn't get updated fast enough. I am wondering if the dashboard is running behind.
The dashboard must display the buy and sell volumes at any moment in time as bar charts. I am also adding some metrics to show the total volume of buy and sell, as well as their change.
Steps to reproduce
My code is structured in the following way.
There is a streamer.py file, that defines a class Streamer. The Streamer object is a Websocket client. It connects to a stream, handles messages, and updates the dashboard. Whenever a new message is received, Streamer acquires a threading.Lock() and updates the pandas dataframes (one dataframe for buy orders and one dataframe for sell orders). If there are multiple orders happening at the same timestamp, it combines them by summing the corresponding volumes. Then, it releases the threading.Lock() and it creates a new thread where the update function (defined in streamer.py) is executed. The update function acquires the lock to avoid messing up with memory.
In the main.py file, streamlit's dashboard and the Streamerobject are initialized.
To reproduce the following code you need to connect to the Websocket from a region where Binance is not restricted. Since I live in the US, I must use a VPN to properly receive the data.
Code snippet:
main.py file
# main.py
import streamer
import pandas as pd
import streamlit as st # web development
import numpy as np # np mean, np random
import time # to simulate a real time data, time loop
import plotly.express as px # interactive charts
df_buy = pd.DataFrame(columns = [ 'Price', 'Quantity', 'USD Value'])
df_sell = pd.DataFrame(columns = [ 'Price', 'Quantity', 'USD Value'])
st.set_page_config(
page_title='Real-Time Data Science Dashboard',
page_icon='✅',
layout='wide'
)
# dashboard title
st.title("Real-Time / Live Data Science Dashboard")
placeholder = st.empty()
streamer.Stream(df_buy,df_sell,placeholder).connect()
streamer.py file
# streamer.py
import websocket
import json
import streamlit as st
import plotly.express as px
import pandas as pd
from threading import Thread, Lock
from streamlit.script_run_context import add_script_run_ctx
from datetime import datetime
import time
def on_close(ws, close_status_code, close_msg):
print('LOG', 'Closed orderbook client')
def update(df_buy,df_sell, placeholder, lock):
lock.acquire()
with placeholder.container():
# create three columns
kpi1, kpi2 = st.columns(2)
current_sumSellVolumes = df_sell['Quantity'].sum()
previous_sumSellVolumes = df_sell.iloc[:-1]['Quantity'].sum()
current_sumBuyVolumes = df_buy['Quantity'].sum()
previous_sumBuyVolumes = df_buy.iloc[:-1]['Quantity'].sum()
# fill in those three columns with respective metrics or KPIs
kpi2.metric(label="Sell quantity 📉", value=round(current_sumSellVolumes, 2),
delta=round(current_sumSellVolumes - previous_sumSellVolumes, 2))
kpi1.metric(label="Buy quantity 📈", value=round(current_sumBuyVolumes, 2),
delta=round(current_sumBuyVolumes - previous_sumBuyVolumes, 2))
# create two columns for charts
fig_col1, fig_col2 = st.columns(2)
with fig_col1:
st.markdown("### Buy Volumes")
fig = px.bar(data_frame=df_buy, x=df_buy.index, y='Quantity')
st.write(fig)
with fig_col2:
st.markdown("### Sell Volumes")
fig2 = px.bar(data_frame=df_sell, x=df_sell.index, y='Quantity')
st.write(fig2)
st.markdown("### Detailed Data View")
st.dataframe(df_buy)
st.dataframe(df_sell)
lock.release()
class Stream():
def __init__(self, df_buy, df_sell, placeholder):
self.symbol = 'BTCUSDT'
self.df_buy = df_buy
self.df_sell = df_sell
self.placeholder = placeholder
self.lock = Lock()
self.url = "wss://stream.binance.com:9443/ws"
self.stream = f"{self.symbol.lower()}#aggTrade"
self.times = []
def on_error(self, ws, error):
print(self.times)
print('ERROR', error)
def on_open(self, ws):
print('LOG', f'Opening WebSocket stream for {self.symbol}')
subscribe_message = {"method": "SUBSCRIBE",
"params": [self.stream],
"id": 1}
ws.send(json.dumps(subscribe_message))
def handle_message(self, message):
self.lock.acquire()
timestamp = datetime.utcfromtimestamp(int(message['T']) / 1000)
price = float(message['p'])
qty = float(message['q'])
USDvalue = price * qty
side = 'BUY' if message['m'] == False else 'SELL'
if side == 'BUY':
df = self.df_buy
else:
df = self.df_sell
if timestamp not in df.index:
df.loc[timestamp] = [price, qty, USDvalue]
else:
df.loc[df.index == timestamp, 'Quantity'] += qty
df.loc[df.index == timestamp, 'USD Value'] += USDvalue
self.lock.release()
def on_message(self, ws, message):
message = json.loads(message)
self.times.append(time.time())
if 'e' in message:
self.handle_message(message)
thr = Thread(target=update, args=(self.df_buy, self.df_sell, self.placeholder, self.lock,))
add_script_run_ctx(thr)
thr.start()
def connect(self):
print('LOG', 'Connecting to websocket')
self.ws = websocket.WebSocketApp(self.url, on_close=on_close, on_error=self.on_error,
on_open=self.on_open, on_message=self.on_message)
self.ws.run_forever()
Debug info
Streamlit version: 1.4.0
Python version: 3.10.4
OS version: MacOS 13.1
Browser version: Safari 16.2

How to run a .py file from another .py file

i am starting with Python and while i was working on a project found this issue:
I have 2 .py files. One that has a Login and one that has a chart built with the Streamlit library.
The thing is that i want to redirect the user to the chart once he logs in. I havent found any response to this issue or i just dont get it.
What i think i have to do is wrap the Streamlit Graph file into a function but i dont know how.
Thanks for anyone reading this.
Login File:
from flask import Flask, render_template, redirect, url_for,request, session
from flask_mysql_connector import MySQL
import pymysql
from templates.home import get_input
app = Flask(__name__)
app.secret_key = "120395"
app.config["MYSQL_HOST"] = "localhost"
app.config["MYSQL_USER"] = "root"
app.config["MYSQL_PASSWORD"] = "anto1203"
app.config["MYSQL_DB"] = "amsterdamdb"
db = MySQL(app)
#app.route('/', methods=['GET', 'POST'])
def index():
if request.method == 'POST':
if ('email_usuario_form' in request.form) and ('password_usuario_form' in request.form):
usuarios_email_py = request.form['email_usuario_form']
usuarios_password_py = request.form['password_usuario_form']
cursor = db.connection.cursor(pymysql.cursors.DictCursor)
cursor.execute("SELECT USUARIOS_EMAIL, USUARIOS_LOGIN_PASSWORD FROM amsterdamdb.usuarios WHERE "
"USUARIOS_EMAIL=%s AND USUARIOS_LOGIN_PASSWORD=%s", (usuarios_email_py,
usuarios_password_py))
info = cursor.fetchone()
print(info)
if info is None:
return "Usuario y contraseña incorrectos"
else:
if info[0] == usuarios_email_py and info[1] == usuarios_password_py:
session['loginsuccess'] = True
return redirect(url_for("home"))
return render_template("login.html")
#app.route('/registro')
def registro():
return render_template("register.html")
#app.route('/home')
def home():
if session['loginsuccess'] == True:
return get_input()
if __name__ == '__main__':
app.run(debug=True)
Streamlit Graph File
import streamlit as st
import pandas as pd
from PIL import Image
from datetime import datetime, timedelta
import json
import requests
import altair as alt
st.write("""
# Bienvenido a Ámsterdam
**Mostrando información del mercado de acciones**
""")
st.sidebar.header('Ingrese parámetros')
def get_input():
aux_datetime = datetime.now()
today_date = aux_datetime.date()
treinta_dias_date = today_date - timedelta(days=30)
get_fecha_desde = st.sidebar.text_input("Fecha Inicial", treinta_dias_date )
get_fecha_hasta = st.sidebar.text_input("Fecha Final", today_date)
get_nemotecnico = st.sidebar.text_input("Identificador", "goog" )
return get_fecha_desde, get_fecha_hasta, get_nemotecnico
def get_data(nemotecnico_empresa, fecha_desde, fecha_hasta):
nemotecnico = nemotecnico_empresa
from_input_fecha_desde,from_input_fecha_hasta,from_input_nemotecnico = get_input()
fecha_desde = datetime.strptime(from_input_fecha_desde, '%Y-%m-%d')
fecha_hasta = datetime.strptime(from_input_fecha_hasta, '%Y-%m-%d')
nemotecnico = from_input_nemotecnico.upper()
fecha_desde_tmsp = str(round(datetime.timestamp(fecha_desde)))
fecha_hasta_tmsp = str(round(datetime.timestamp(fecha_hasta)))
if nemotecnico=="":
r = requests.get(
'https://finnhub.io/api/v1/stock/candle?symbol=&resolution=1&from=0&to=0&token=btagn3v48v6vivh8p9n0')
accionesJson = r.json()
else:
r = requests.get(
'https://finnhub.io/api/v1/stock/candle?symbol=' + nemotecnico + '&resolution=D&from=' + fecha_desde_tmsp + '&to=' + fecha_hasta_tmsp + '&token=wathever')
accionesJson = r.json()
open_values = accionesJson["o"]
close_values = accionesJson["c"]
fecha_values = accionesJson["t"]
fecha_values_size = len(fecha_values)
fecha_values_int = []
fecha_values_datetime = []
for x in range(0, fecha_values_size):
fecha_values_int.append(int(fecha_values[x]))
for i in range(0, fecha_values_size):
valor_aux = datetime.fromtimestamp((fecha_values_int[i]))
fecha_values_datetime.append(valor_aux.strftime("%m/%d/%Y"))
data_ordenada = {'Apertura': open_values,
'Cierre': close_values,
'Fecha': fecha_values_datetime}
chart_data = pd.DataFrame(data_ordenada)
source = chart_data
alt_chart = alt.Chart(source).transform_fold(
['Apertura', 'Cierre']
).mark_line().encode(
x=alt.X('Fecha:O',axis=alt.Axis(title="Fechas")),
y=alt.Y('value:Q',scale=alt.Scale(zero=False), axis=alt.Axis(title="Valor")),
tooltip=['Fecha', 'Apertura', 'Cierre'],
color='key:N'
).interactive().properties(
width=800,
height=500
)
st.write("", "", alt_chart)
st.header('Puntos de interés')
st.write(source.describe())
print(accionesJson)

Returning matplotlib plots using telegram bot

This code is from here
I have the following code for a telegram bot which i am building:
import pandas as pd
from pandas import datetime
from pandas import DataFrame as df
import matplotlib
from pandas_datareader import data as web
import matplotlib.pyplot as plt
import datetime
import requests
from bottle import (
run, post, response, request as bottle_request
)
BOT_URL = 'https://api.telegram.org/bot128secretns/'
def get_chat_id(data):
"""
Method to extract chat id from telegram request.
"""
chat_id = data['message']['chat']['id']
return chat_id
def get_message(data):
"""
Method to extract message id from telegram request.
"""
message_text = data['message']['text']
return message_text
def send_message(prepared_data):
"""
Prepared data should be json which includes at least `chat_id` and `text`
"""
message_url = BOT_URL + 'sendMessage'
requests.post(message_url, json=prepared_data)
def get_ticker(text):
stock = f'^GSPC'
start = datetime.date(2000,1,1)
end = datetime.date.today()
data = web.DataReader(stock, 'yahoo',start, end)
plot = data.plot(y='Open')
return plot
def prepare_data_for_answer(data):
answer = get_ticker(get_message(data))
json_data = {
"chat_id": get_chat_id(data),
"text": answer,
}
return json_data
#post('/')
def main():
data = bottle_request.json
answer_data = prepare_data_for_answer(data)
send_message(answer_data) # <--- function for sending answer
return response # status 200 OK by default
if __name__ == '__main__':
run(host='localhost', port=8080, debug=True)
When i run this code i am getting the following error:
TypeError: Object of type AxesSubplot is not JSON serializable
What this code is suppose to do is take ticker symbols from telegram app and return its chart back.
I know this is because json does not handle images.
What can i do to resolve it?

Sorry, I'm a bit late to the party. Here is a possible solution below, though I didn't test it. Hope it works or at least gives you a way to go about solving the issue :)
import datetime
from io import BytesIO
import requests
from pandas_datareader import data as web
from bottle import (
run, post, response, request as bottle_request
)
BOT_URL = 'https://api.telegram.org/bot128secretns/'
def get_chat_id(data):
"""
Method to extract chat id from telegram request.
"""
chat_id = data['message']['chat']['id']
return chat_id
def get_message(data):
"""
Method to extract message id from telegram request.
"""
message_text = data['message']['text']
return message_text
def send_photo(prepared_data):
"""
Prepared data should be json which includes at least `chat_id` and `plot_file`
"""
data = {'chat_id': prepared_data['chat_id']}
files = {'photo': prepared_data['plot_file']}
requests.post(BOT_URL + 'sendPhoto', json=data, files=files)
def get_ticker(text):
stock = f'^GSPC'
start = datetime.date(2000,1,1)
end = datetime.date.today()
data = web.DataReader(stock, 'yahoo',start, end)
plot = data.plot(y='Open')
return plot
def prepare_data_for_answer(data):
plot = get_ticker(get_message(data))
# Write the plot Figure to a file-like bytes object:
plot_file = BytesIO()
fig = plot.get_figure()
fig.savefig(plot_file, format='png')
plot_file.seek(0)
prepared_data = {
"chat_id": get_chat_id(data),
"plot_file": plot_file,
}
return prepared_data
#post('/')
def main():
data = bottle_request.json
answer_data = prepare_data_for_answer(data)
send_photo(answer_data) # <--- function for sending answer
return response # status 200 OK by default
if __name__ == '__main__':
run(host='localhost', port=8080, debug=True)
The idea is not to send a message using the sendMessage Telegram API endpoint, but to send a photo file by using the sendPhoto endpoint. Here, we use savefig call in the prepare_data_for_answer function body to convert AxesSubplot instance, that we get as a return value from the get_ticker function, to a file-like BytesIO object, which we then send as a photo to Telegram using send_photo function (previously named as send_message).

You may use bob-telegram-tools
from bob_telegram_tools.bot
import TelegramBot
import matplotlib.pyplot as plt
token = '<your_token>'
user_id = int('<your_chat_id>')
bot = TelegramBot(token, user_id)
plt.plot([1, 2, 3, 4])
plt.ylabel('some numbers')
bot.send_plot(plt)
# This method delete the generetad image
bot.clean_tmp_dir()

You cannot send a matplotlib figure directly. You will need to convert it to bytes and then send it as a multipart message.
data.plot will return a matplotlib.axes.Axes object. You can save convert the figure to bytes like this
import StringIO
img = StringIO.StringIO()
plot.fig.savefig(img, format='png')
img.seek(0)
yukuku/telebot has some good code on how to send the image as a message. Check this line here.

Resampling Live Websocket Ticks to Candles using Pandas in python

I am trying to resample live ticks from KiteTicker websocket into OHLC candles using pandas and this is the code I have written, which works fine with single instrument (The commented trd_portfolio on line 9) but doesn't work with multiple instruments (Line 8) as it mixes up data of different instruments.
Is there any way to relate the final candles df to instrument tokens? or make this work with multiple intruments?
I would like to run my algo on multiple instruments at once, please suggest if there is a better way around it.
from kiteconnect import KiteTicker;
from kiteconnect import KiteConnect;
import logging
import time,os,datetime,math;
import winsound
import pandas as pd
trd_portfolio = {954883:"USDINR19MARFUT",4632577:"JUBLFOOD"}
# trd_portfolio = {954883:"USDINR19MARFUT"}
trd_tkn1 = [];
for x in trd_portfolio:
trd_tkn1.append(x)
c_id = '****************'
ak = '************'
asecret = '*************************'
kite = KiteConnect(api_key=ak)
print('[*] Generate access Token : ',kite.login_url())
request_tkn = input('[*] Enter Your Request Token Here : ')[-32:];
data = kite.generate_session(request_tkn, api_secret=asecret)
kite.set_access_token(data['access_token'])
kws = KiteTicker(ak, data['access_token'])
#columns in data frame
df_cols = ["Timestamp", "Token", "LTP"]
data_frame = pd.DataFrame(data=[],columns=df_cols, index=[])
def on_ticks(ws, ticks):
global data_frame, df_cols
data = dict()
for company_data in ticks:
token = company_data["instrument_token"]
ltp = company_data["last_price"]
timestamp = company_data['timestamp']
data[timestamp] = [timestamp, token, ltp]
tick_df = pd.DataFrame(data.values(), columns=df_cols, index=data.keys()) #
data_frame = data_frame.append(tick_df)
ggframe=data_frame.set_index(['Timestamp'],['Token'])
print ggframe
gticks=ggframe.ix[:,['LTP']]
candles=gticks['LTP'].resample('1min').ohlc().dropna()
print candles
def on_connect(kws , response):
print('Connected')
kws.subscribe(trd_tkn1)
kws.set_mode(kws.MODE_FULL, trd_tkn1)
def on_close(ws, code, reason):
print('Connection Error')
kws.on_ticks = on_ticks
kws.on_connect = on_connect
kws.on_close = on_close
kws.connect()

I don't have access to the Kite API, but I've been looking at some code snippets that use it trying to figure out another issue I'm having related to websockets. I came across this open question, and I think I can help, though I can't really test this solution.
The problem I think is that you're not calculating OHLC for each "token"... it just does it for all tokens.
data_frame = data_frame.append(tick_df)
ggframe=data_frame.set_index('Timestamp')
candles=ggframe.groupby('token').resample('1min').agg({'LTP':'ohlc'})
You'll get a multi-index output, but the column names might not quite line up for the rest of your code. To fix that:
candles.columns=['open','high','low','close']

Create a graph from a CSV file and render to browser with Django and the Pandas Python library

I'm learning how to use the Django framework for a work project that will allow users to load files in various formats (at the moment I am only dealing with CSV files), graph that data using Pandas, and display that data back to the user via a Django template. I haven't had any problems creating the graph in iPython, but have been struggling with getting it to an HTML Django template.
I've followed the following example from matplotlib:
# graph input file
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
from matplotlib.figure import Figure
from matplotlib.dates import DateFormatter
def graph(request):
fig = Figure()
ax = fig.add_subplot(111)
x = []
y = []
now = datetime.datetime.now()
delta = datetime.timedelta(days=1)
for i in range(10):
x.append(now)
now += delta
y.append(random.randint(0, 1000))
ax.plot_date(x, y, '-')
ax.xaxis.set_major_formatter(DateFormatter('%Y-%m-%d'))
fig.autofmt_xdate()
canvas = FigureCanvas(fig)
response = HttpResponse( content_type = 'image/png')
canvas.print_png(response)
return response
The above example works great and I can see it in a template, but that's just a graph with hard-coded values.
I've attempted to use Pandas because of its seemingly simplistic syntax and my attempts in Django are as follows:
# graph input file
import pandas as pd
from pandas import DataFrame
def graph(request):
data_df = pd.read_csv("C:/Users/vut46744/Desktop/graphite_project/sampleCSV.csv")
data_df = pd.DataFrame(dataArray)
data_df.plot()
response = HttpResponse( content_type = 'image/png')
return response
In Django calling the .plot() displays the graph fine, but displays a blank page to the HTML template. I've also tried using Numpy's genfromtxt() and loadtxt(), but to no avail. Also, my Google searches have not been fruitful either.
Any help or suggestion would be great. If you know of a better alternative to Pandas then I am willing to try other options.

Haven't tried this yet, but I would attack it something like:
def graph(request):
fig = Figure()
ax = fig.add_subplot(111)
data_df = pd.read_csv("C:/Users/vut46744/Desktop/graphite_project/sampleCSV.csv")
data_df = pd.DataFrame(data_df)
data_df.plot(ax=ax)
canvas = FigureCanvas(fig)
response = HttpResponse(content_type='image/png')
canvas.print_png(response)
return response