I have a Redis pub-sub channel 'price-updates' in redis for which a publisher sets updates for a stock price. I want to display a streaming grid that keeps appending the price updates as they come at the end of the grid.
So far, I have created a non-working version of what I want to do.
from streamz import Stream
from streamz.dataframe import DataFrame
source = Stream()
data = []
def handler(message):
json_data = json.loads(message['data'])
df = pd.DataFrame.from_dict([json_data]).set_index('sym')
source.map(handler).sink(data.append)
sdf = DataFrame(source)
## Run this in a different thread
p.subscribe('price-updates')
while True:
message = p.get_message()
if message:
source.emit(message)
time.sleep(0.001)
## end of thread block
#displayStreamingDataGrid(sdf)
I would appreciate if someone with more experience with the sdf could help me do this.
I was able to do this without streams. However, I am not getting a the streaming grid that gets updated in-place but rather displaying and clearing the output which is very annoying.
Here is the output widget in one jupyter cell
import ipywidgets as iw
from IPython.display import display
o = iw.Output()
def output_to_widget(df, output_widget):
output_widget.clear_output()
with output_widget:
display(df)
o
Here is the code to subscribe to redis and get handle the message
import redis, json, time
r = redis.StrictRedis(host = HOST, password = PASS, port = PORT, db = DB)
p = r.pubsub(ignore_subscribe_messages=True)
p.subscribe('QUOTES')
mdf = pd.DataFrame()
while True:
message = p.get_message()
if message:
json_msg = json.loads(message['data'])
df = pd.DataFrame([json_msg]).set_index('sym')
mdf = mdf.append(df)
output_to_widget(mdf, o)
time.sleep(0.001)
You can use https://github.com/AaronWatters/jp_proxy_widget to create an html
table which you can update in place without visibly clearing the table between updates.
I put an example notebook here: https://github.com/AaronWatters/jp_doodle/blob/master/notebooks/misc/In%20place%20html%20table%20update%20demo.ipynb
The trick is to create a widget that displays a table and attaches
an update operation which modifies the table:
# Create a proxy widget with a table update method
import jp_proxy_widget
def updateable_table(headers, rows):
w = jp_proxy_widget.JSProxyWidget()
w.js_init("""
# injected javascript for the widget:
element.update_table = function(headers, rows) {
element.empty();
var table = $("<table border style='text-align:center'/>");
table.appendTo(element);
var header_row = $("<tr/>");
for (var i=0; i<headers.length; i++) {
$("<th style='text-align:center'>" + headers[i] + "</th>")
.width(50)
.appendTo(header_row);
}
header_row.appendTo(table);
for (var j=0; j<rows.length; j++) {
var table_row = $("<tr/>").appendTo(table);
var data_row = rows[j];
for (var i=0; i<data_row.length; i++) {
$("<td>" + data_row[i] + "</td>").appendTo(table_row);
}
}
}
element.update_table(headers, rows);
""", headers=headers, rows=rows)
return w
# show the widget
w = updateable_table(headers, rows)
w
The code to update the widget
# Update the widget 20 times
import time
count = -20
for i in range(21):
time.sleep(1)
rows = [rows[-1]] + rows[:-1] # rotate the rows
rows[0][0] = count # change the upper left entry.
count += 1
w.element.update_table(headers, rows)
updates the table in place with no visible erasure. The example
notebook linked above also shows how to do the same thing using a
pandas dataframe.
Related
In my code I am trying to update a table which is called bag_table (in the row of the container of the right column). And when running my code it does show the table initially, but when I hit the submit button the backed is working. It is actually updating the bag_table variable, but it does not update in the gui itself.
Below is my full code.
pd.options.display.max_columns = 100
from services.bag import PCHN
from utils.convertors import dataframe_to_datatable
import flet as ft
def main(page: ft.page):
def bag_service(e):
pc = '9722LA' if postal_code_field.value == '' else postal_code_field.value
hn = '29' if house_number_field.value == '' else house_number_field.value
address = PCHN(pc,
hn).result
bag_table = dataframe_to_datatable(address)
page.add(bag_table) # I added this for debugging, it is actually adding the table at the bottom of my page, so it is updating the actual bag_table
page.update() # This is not updating my bag_table in place though. It stays static as it is.
# define form fields
postal_code_field = ft.TextField(label='Postal code')
house_number_field = ft.TextField(label='House number')
submit_button = ft.ElevatedButton(text='Submit', on_click=bag_service)
# fields for the right column
address = PCHN('9722GN', '5').result
bag_table = dataframe_to_datatable(address)
# design layout
# 1 column to the left as a frame and one to the right with two rows
horizontal_divider = ft.Row
left_column = ft.Column
right_column = ft.Column
# fill the design
page.add(
horizontal_divider(
[left_column(
[postal_code_field,
house_number_field,
submit_button
]
),
right_column(
[
ft.Container(
ft.Row([bag_table],
scroll='always'),
bgcolor=ft.colors.BLACK,
width=800
)
]
)
]
)
)
if __name__ == '__main__':
ft.app(target=main,
view=ft.WEB_BROWSER,
port=666
)
I have been trouble shooting this like craze (hence all the print statements), but it's a classical thing of looking at a thing for hours, and it's probably a stupid mistake. Any help would be much appreciated.
This is a classic "python variables are just references, not values" problem.
You need something that references the table, but isn't the table itself.
Fortunately you already use a container here:
ft.Row([bag_table],
So we can take advantage of that.
In your setup where you create the original table you also need a container:
bag_table = dataframe_to_datatable(address)
bag_container = [bag_table]
and
...
ft.Container(
ft.Row(bag_container,
scroll='always'),
bgcolor=ft.colors.BLACK,
width=800
)
...
Now you can change what bag_container contains:
def bag_service(e):
pc = '9722LA' if postal_code_field.value == '' else postal_code_field.value
hn = '29' if house_number_field.value == '' else house_number_field.value
address = PCHN(pc,
hn).result
bag_container[0] = dataframe_to_datatable(address)
page.update()
I’m using this code to draw myself a server in visio:
import win32com.client as w32
visio = w32.Dispatch("visio.Application")
visio.Visible = 1
doc = visio.Documents.Add("Detailed Network Diagram.vst")
page = doc.Pages.Item(1)
page.name = "My drawing"
stn2 = visio.Documents("Servers.vss")
server = stn2.Masters("Server")
serv = page.Drop(server, 0, 0)
for ssh in serv.shapes:
ssh.Cells( 'Fillforegnd' ).FormulaForceU = 'RGB(255,0,0)'
And my problem is when I’m trying to fill the object with a color (instead of the regular server color) it doesn’t work.
Nothing really worked. I’m using python 3.8.
Try this code please
import win32com.client as w32
visio = w32.Dispatch("visio.Application")
visio.Visible = 1
doc = visio.activedocument
page = doc.pages(1)
page.name = "Mydrawing"
stn2 = visio.Documents(2)
server = stn2.Masters(2)
serv = page.Drop(server, 0, 0)
#iterate all sub-shapes into Serv-shape
for ssh in serv.shapes:
ssh.Cells( 'Fillforegnd' ).FormulaForceU = 'RGB(255,255,0)'
If you dont need iterate all sub-shapes, you can change only same of them
#iterate 2nd, 3rd and 4rd sub-shapes into Serv-shape #
for i in range (2,5):
ssh = serv.shapes(i)
# if you need get solid color for sub-shapes uncomment next line
# ssh.Cells('FillPattern').FormulaForceU = '1'
ssh.Cells('Fillforegnd').FormulaU = 'RGB(255,255,0)'
Code in my Jupyterlab notebook change only 3 sub-shapes, which i select and delete for demonstrate difference…
PS The user's problem was not in the code, but in the Visio sub-shapes, which did not want to inherit the color of the main shape. Because these sub-shapes had formulas that used functions like THEMEGUARD and similar to it in their cells.
I modified the shape from the built-in set of elements and the problem was solved…
PPS Solved! To remove the dependency on those sub-shapes, you need to change their Fillstyle to Normal. Just add new line of code ssh.FillStyle = 'Normal'.
Look at code ↓
import win32com.client as w32
visio = w32.Dispatch("visio.Application")
visio.Visible = 1
# create document based on Detailed Network Diagram template (use full path)
doc = visio.Documents.Add ("C:\Program Files\Microsoft Office\root\Office16\visio content\1033\dtlnet_m.vstx")
# use one of docked stencils
stn2 = visio.Documents("PERIPH_M.vssx")
# define 'Server' master-shape
server = stn2.Masters("Server")
# define page
page = doc.Pages.Item(1)
# rename page
page.name = "My drawing"
# drop master-shape on page, define 'Server' instance
serv = page.Drop(server, 0, 0)
# iterate sub-shapes (side edges)
for i in range (2,6):
# define one od side edges from 'Server'
ssh = serv.shapes(i)
# Change Fill Style to 'Normal'
ssh.FillStyle = 'Normal'
# fix FillForegnd cell for side edge
ssh.Cells( 'Fillforegnd' ).FormulaForceU = 'Guard(Sheet.' + str(serv.id) + '!FillForegnd)'
# fix FillBkgnd cell for side edge
ssh.Cells( 'FillBkgnd' ).FormulaForceU = 'Guard(Sheet.' + str(serv.id) + '!FillBkgnd)'
# instead formula 'Guard(x)' rewrite formula 'Guard(1)'
ssh.Cells( 'FillPattern' ).FormulaForceU = 'Guard(1)'
# fill main shape in 'Server' master-shape
serv.Cells("FillForegnd").FormulaForceU = '5'
I am trying to resample live ticks from KiteTicker websocket into OHLC candles using pandas and this is the code I have written, which works fine with single instrument (The commented trd_portfolio on line 9) but doesn't work with multiple instruments (Line 8) as it mixes up data of different instruments.
Is there any way to relate the final candles df to instrument tokens? or make this work with multiple intruments?
I would like to run my algo on multiple instruments at once, please suggest if there is a better way around it.
from kiteconnect import KiteTicker;
from kiteconnect import KiteConnect;
import logging
import time,os,datetime,math;
import winsound
import pandas as pd
trd_portfolio = {954883:"USDINR19MARFUT",4632577:"JUBLFOOD"}
# trd_portfolio = {954883:"USDINR19MARFUT"}
trd_tkn1 = [];
for x in trd_portfolio:
trd_tkn1.append(x)
c_id = '****************'
ak = '************'
asecret = '*************************'
kite = KiteConnect(api_key=ak)
print('[*] Generate access Token : ',kite.login_url())
request_tkn = input('[*] Enter Your Request Token Here : ')[-32:];
data = kite.generate_session(request_tkn, api_secret=asecret)
kite.set_access_token(data['access_token'])
kws = KiteTicker(ak, data['access_token'])
#columns in data frame
df_cols = ["Timestamp", "Token", "LTP"]
data_frame = pd.DataFrame(data=[],columns=df_cols, index=[])
def on_ticks(ws, ticks):
global data_frame, df_cols
data = dict()
for company_data in ticks:
token = company_data["instrument_token"]
ltp = company_data["last_price"]
timestamp = company_data['timestamp']
data[timestamp] = [timestamp, token, ltp]
tick_df = pd.DataFrame(data.values(), columns=df_cols, index=data.keys()) #
data_frame = data_frame.append(tick_df)
ggframe=data_frame.set_index(['Timestamp'],['Token'])
print ggframe
gticks=ggframe.ix[:,['LTP']]
candles=gticks['LTP'].resample('1min').ohlc().dropna()
print candles
def on_connect(kws , response):
print('Connected')
kws.subscribe(trd_tkn1)
kws.set_mode(kws.MODE_FULL, trd_tkn1)
def on_close(ws, code, reason):
print('Connection Error')
kws.on_ticks = on_ticks
kws.on_connect = on_connect
kws.on_close = on_close
kws.connect()
I don't have access to the Kite API, but I've been looking at some code snippets that use it trying to figure out another issue I'm having related to websockets. I came across this open question, and I think I can help, though I can't really test this solution.
The problem I think is that you're not calculating OHLC for each "token"... it just does it for all tokens.
data_frame = data_frame.append(tick_df)
ggframe=data_frame.set_index('Timestamp')
candles=ggframe.groupby('token').resample('1min').agg({'LTP':'ohlc'})
You'll get a multi-index output, but the column names might not quite line up for the rest of your code. To fix that:
candles.columns=['open','high','low','close']
I am trying to display live price updates coming from a redis pubsub channel in a grid in Jupyter. Everytime there is a price update, the message will be added at the end of the grid. In order words, a gridview widget will be tied to a Dataframe so everytime it changes, the gridview will change. The idea is to get something like this:
I tried to do that by displaying and clearing the output. However, I am not getting a the streaming grid that gets updated in-place but rather displaying and clearing the output which is very annoying.
Here is the output widget in one jupyter cell
import ipywidgets as iw
from IPython.display import display
o = iw.Output()
def output_to_widget(df, output_widget):
output_widget.clear_output()
with output_widget:
display(df)
o
Here is the code to subscribe to redis and get handle the message
import redis, json, time
r = redis.StrictRedis(host = HOST, password = PASS, port = PORT, db = DB)
p = r.pubsub(ignore_subscribe_messages=True)
p.subscribe('QUOTES')
mdf = pd.DataFrame()
while True:
message = p.get_message()
if message:
json_msg = json.loads(message['data'])
df = pd.DataFrame([json_msg]).set_index('sym')
mdf = mdf.append(df)
output_to_widget(mdf, o)
time.sleep(0.001)
Try changing the first line of output_to_widget to output_widget.clear_output(wait = True).
https://ipython.org/ipython-doc/3/api/generated/IPython.display.html
I was able to get it to work using Streaming DataFrames from the streamz library.
Here is the class to emit the data to the streamming dataframe.
class DataEmitter:
def __init__(self, pubsub, src):
self.pubsub = pubsub
self.src = src
self.thread = None
def emit_data(self, channel):
self.pubsub.subscribe(**{channel: self._handler})
self.thread = self.pubsub.run_in_thread(sleep_time=0.001)
def stop(self):
self.pubsub.unsubscribe()
self.thread.stop()
def _handler(self, message):
json_msg = json.loads(message['data'])
df = pd.DataFrame([json_msg])
self.src.emit(df)
and here is the cell to display the streaming dataframe
r = redis.StrictRedis(host = HOST, password = PASS, port = PORT, db = DB)
p = r.pubsub(ignore_subscribe_messages=True)
source = Stream()
emitter = DataEmitter(p, source, COLUMNS)
emitter.emit_data(src='PRICE_UPDATES')
#sample for how the dataframe it's going to look like
example = pd.DataFrame({'time': [], 'sym': []})
sdf = source.to_dataframe(example=example)
sdf
so i'm trying to display an R generated image while using wxpython and rpy2...
The full 131 line code is here... https://gist.github.com/ACollectionOfAtoms/4286c0fc32838b03f2ea
So within the program, the user comes to a point where they are a window genereated by lst_view, which has two buttons "Ok" and "Visualize". Once Visualize is pressed this code is executed..
def graph(self,event):
f = open('results.csv', 'wb')
csvwriter = csv.writer(f)
for i in self.res:
for j in range(1,len(i)):
row = [i[0] ,str(i[j])]
csvwriter.writerow(row)
f.close()
r = robjects.r
r('''
source('vis.r')
''')
r_main = robjects.globalenv['main']
r_main()
return True
Where in vis.r we have:
graph <- function() {
res = read.csv("results.csv", header=FALSE)
res = mutate(res, Percent = 100*(V2/(sum(res$V2))))
ggplot(data=res, aes(x=V1, y=V2, fill=Percent)) + geom_bar(stat="identity") + coord_flip() + xlab('Facility') + ylab('Number Of Violations')
}
main <- function(){
print(graph())
}
This doesn't immediately generate the graph, instead it causes a new menu to appear and the graphic only displays if I go to "Page Setup"....
Anyone ideas?!
Alright, thanks to unutbu I was able to figure this out!
Basically I initially wanted to save the image and then visualize it with wx, but i had problems using dev.off() and such through rpy2!
BUT! unutbu provided a link and saved me a ton of trouble. you can use ggsave() to completely circumvent the whole jpeg('this.jpg'); dev.off(); business!
Thanks again, unutbu. Now i'll just render it with wx no problem!