Sorry I cannot give all the code.
Basically, I am retrieving the OrderBook of a few hundred cryptocurrencies every second.
Every time I retrieve it I need to add it into an excel readable file for another department.
Every excel will save all the data in one day.
The retrieved order book record will be something that looks like this.
time | exchange | price | quantity | side
unit time| BINANCE | 1.00 | 90925 | ask
I have tried pd.excelwriter append, read_csv combine write_csv and appending to pre-saved pandaframe.
However, all these options are too slow to get under 1 second even I boosted the code with cython.
Websocket with thread only takes 0.2s to retrieve the data so we have 0.8s to append.
Something is fixed: Python should be the language, my co-worker only knows python, the saving format must be any file extension that excel can read as the other department doesn't know coding.
Can someone help me with some ideas? I can do the implementation myself. Just want some idea.
A slightly different approach could be to write an Excel add-in in Python to get the data into Excel. That should be much faster than writing out a workbook each time the data changes.
You can do that using PyXLL (https://www.pyxll.com), which is a commercial (paid for) product. Using PyXLL you could write RTD (real time data) functions to stream real time data directly into Excel.
Your colleagues would also need the PyXLL add-in installed, configured to load your Python code - then they would be able to access your functions and macros etc to get the real time data. PyXLL is commonly used in finance for this type of application so it might just be what you're looking for...
This post shows how to get real time prices in Excel from BitMEX. I know you're using Binance but it might be interesting for you anyway :) https://towardsdatascience.com/live-streaming-crypto-prices-in-excel-aaa41628bc53
Related
All,
I am trying to look at trends over time with my data. However, all of the data I have is static (snapshot in time). Therefore, I do not have any way to track how long something has been in inventory for. I know that I could export the data, and then save it to a table for the report to reference, but that is manual and manual tasks introduce errors. Therefore, I am looking to automate it. I am experienced in VBA but new to Python, so would this be possible by creating a python scrip to write a CSV file with a date timestamp and then pull that CSV file back into the Power bi report?
If this is possible to do without coding, that would be great. I am open to any and all ideas.
Thanks,
Brian
Thanks for taking the time to read my question.
I am working on a personal project to learn python scripting for excel, and I want to learn how to move data from one workbook to another.
In this example, I am emulating a company employee ledger that has name, position, address, and more (The organizations is by row so every employee takes up one row). But the project is to have a selected number of people be transferred to a new ledger (another excel file). So I have a list of emails in a .txt file (it could even be another excel file but I thought .txt would be easier), and I would want the script to run through the .txt file, get the emails, and look for any rows that have a matching email address(all emails are in cell 'B'). And if any are found, then copy that entire row to the new excel file.
I tried a lot of ways to make this work, but I could not figure it out. I am really new to python so I am not even sure if this is possible. Would really appreciate some help!
You have essentially two packages that will allow manipulation of Excel files. For reading in data and performing analysis the standard package for use is pandas. You can save the files as .xlsx however you are only really working with base table data and not the file itself (IE, you are extracing data FROM the file, not working WITH the file)
However what you need is really to perform manipulation on Excel files directly which is better done with openpyxl
You can also read files (such as your text file) using with open function that is native to Python and is not a third party import like pandas or openpyxl.
Part of learning to program includes learning how to use documentation.
As such, here is the documentation you require with sufficient examples to learn openpyxl: https://openpyxl.readthedocs.io/en/stable/
And you can learn about pandas here: https://pandas.pydata.org/docs/user_guide/index.html
And you can learn about python with open here: https://docs.python.org/3/tutorial/inputoutput.html
Hope this helps.
EDIT: It's possible I or another person can give you a specific example using your data / code etc, but you would have to provide it fully. Since you're learning, I suggest using the documentation or youtube.
I've been asked to create a Python script to automate a server deployment for 80 retail stores.
As part of this script, I have a secondary script that I call to change multiple values in 9 XML files, however, the values are unique for each store, so this script needs to be changed each time, but after I am gone, this is going to be done by semi / non-technical people, so we don't want them to change the Python scripts directly for fear of breaking them.
This in mind, I would like to have these people input the store details into an XLS sheet, and a python file read this sheet and put the data it finds into the existing python script with the data to be changed.
The file will be 2 columns, with the required data in the 2nd one.
I'm sorry if this is a long explanation, but that is the gist of it. I'm using python 2.6. Does anyone have a clue about how I can do this? Or which language might be better for this. I also know Bash and Javascript.
Thanks in advance
Depending on the complexity and the volume of your data
for small Openpyxl,
for large pandas
I'm gonna use data from a .csv to train a model to predict user activity on google ads (impressions, clicks) in relation to the weather for a given day. And I have a .csv that contains 6000+ recordings of this info and want to parse it into a database using Python.
I tried making a df in pandas but for some reason the whole table isn't shown. The middle columns (there's about 7 columns I think) and rows (numbered over 6000 as I mentioned) are replaced with '...' when I print the table so I'm not sure if the entirety of the information is being stored and if this will be usable.
My next attempt will possible be SQLite but since it's local memory, will this interfere with someone else making requests to my API endpoint if I don't have the db actively open at all times?
Thanks in advance.
If you used pd.read_csv() i can assure you all of the info is there, it's just not displaying it.
You can check by doing something like print(df['Column_name_you_are_interested_in'].tolist()) just to make sure though. You can also use the various count type methods in pandas to make sure all of your lines are there.
Panadas is pretty versatile so it shouldn't have trouble with 6000 lines
I am new to python. I want to create one excel sheet which contains all the employee weekly efforts. For that, employees need to fill the form that input will directly store in the excel sheet. Once everyone is finished I need to get that automatically filled excel sheet. Is it possible in python? Please can someone guide me on this how to proceed...
Yeah, try to show some code to prove you've tried first.
You could start will building the data structure you want for your application. Like
Name | Hours Spent for this week | Current Time
Then, you can use
import csv
to create a csv file to do so.
CSV module is impleted in python, there is also an excel one, but I highly recommend CSV, because it is easier to be processed by other programs.