All,
I am trying to look at trends over time with my data. However, all of the data I have is static (snapshot in time). Therefore, I do not have any way to track how long something has been in inventory for. I know that I could export the data, and then save it to a table for the report to reference, but that is manual and manual tasks introduce errors. Therefore, I am looking to automate it. I am experienced in VBA but new to Python, so would this be possible by creating a python scrip to write a CSV file with a date timestamp and then pull that CSV file back into the Power bi report?
If this is possible to do without coding, that would be great. I am open to any and all ideas.
Thanks,
Brian
Related
Sorry I cannot give all the code.
Basically, I am retrieving the OrderBook of a few hundred cryptocurrencies every second.
Every time I retrieve it I need to add it into an excel readable file for another department.
Every excel will save all the data in one day.
The retrieved order book record will be something that looks like this.
time | exchange | price | quantity | side
unit time| BINANCE | 1.00 | 90925 | ask
I have tried pd.excelwriter append, read_csv combine write_csv and appending to pre-saved pandaframe.
However, all these options are too slow to get under 1 second even I boosted the code with cython.
Websocket with thread only takes 0.2s to retrieve the data so we have 0.8s to append.
Something is fixed: Python should be the language, my co-worker only knows python, the saving format must be any file extension that excel can read as the other department doesn't know coding.
Can someone help me with some ideas? I can do the implementation myself. Just want some idea.
A slightly different approach could be to write an Excel add-in in Python to get the data into Excel. That should be much faster than writing out a workbook each time the data changes.
You can do that using PyXLL (https://www.pyxll.com), which is a commercial (paid for) product. Using PyXLL you could write RTD (real time data) functions to stream real time data directly into Excel.
Your colleagues would also need the PyXLL add-in installed, configured to load your Python code - then they would be able to access your functions and macros etc to get the real time data. PyXLL is commonly used in finance for this type of application so it might just be what you're looking for...
This post shows how to get real time prices in Excel from BitMEX. I know you're using Binance but it might be interesting for you anyway :) https://towardsdatascience.com/live-streaming-crypto-prices-in-excel-aaa41628bc53
#Background
I am currently playing with some web scraping project as I am learning python.
I have a project which scrapes products with information about price etc using Selenium.
Than I add every record to pandas DF, do some additional data manipulation and than store data in csv and upload to google drive. This runs every night
#Question itself
I would like to watch price changes, new products etc. Would you recommend, how to store data with date key, so there is option to flag new products etc?
My idea is to store every load in one csv and add one column with "date_of_load"... But this seems noob_like... Maybe store data in PostrgreDB? I would like to start learning SQL, so I would try making my own DB.
Thanks for your ideas
As for me better to use NoSQL (Mongo) for this task. You can create JSON (data of prices) with keys are date.
This can help you:
https://www.mongodb.com/blog/post/getting-started-with-python-and-mongodb
https://www.mongodb.com/python
https://realpython.com/introduction-to-mongodb-and-python/
https://www.google.com/search?&q=python+mongo
That is cool! I would suggest sqlite3 (https://docs.python.org/3/library/sqlite3.html) just to get a feeling with SQL. As you can see, it says "It’s also possible to prototype an application using SQLite and then port the code to a larger database such as PostgreSQL or Oracle", which is sort of what you suggested(?), so it could be a nice place to start.
However, CSV might do just fine. As long as there is not too much data (it takes forever to load(and process) all your necessary data), it doesn't matter much how you store it as long as you manage to apply it as you desire.
I've been asked to create a Python script to automate a server deployment for 80 retail stores.
As part of this script, I have a secondary script that I call to change multiple values in 9 XML files, however, the values are unique for each store, so this script needs to be changed each time, but after I am gone, this is going to be done by semi / non-technical people, so we don't want them to change the Python scripts directly for fear of breaking them.
This in mind, I would like to have these people input the store details into an XLS sheet, and a python file read this sheet and put the data it finds into the existing python script with the data to be changed.
The file will be 2 columns, with the required data in the 2nd one.
I'm sorry if this is a long explanation, but that is the gist of it. I'm using python 2.6. Does anyone have a clue about how I can do this? Or which language might be better for this. I also know Bash and Javascript.
Thanks in advance
Depending on the complexity and the volume of your data
for small Openpyxl,
for large pandas
I am working on a project with various sensors. I will be sending data from the sensor system at midnight each night. I want the data from that day only to be added. Here is the problem: The sensors are running 24/7 and are outputting data into csv files that are stored on a Raspberry-Pi. Some are outputting data every 30 minutes, some every hour, and some every 5 minutes.
My question: How do I, via python, ensure that the dataframes that are inputted into the database reflect only data from that day? I understand some of the syntax from mysql that would allow this, such as select(MAX from...) but that I think doesn't apply here.
Any help would be greatly appreciated.
Nevermind I found a method, I will post it here because I feel like it's a little bit of a dodgy way of doing it.
#Use
df.to_sql(name='tdr',con=engine,if_exists='replace',index=False)
#Instead of
df.to_sql(name='tdr',con=engine,if_exists='append',index=False)
This replaces all of the data in said file, and repopulates it with the previous sensor data as well as the new rows in the csv file.
I am new to python. I want to create one excel sheet which contains all the employee weekly efforts. For that, employees need to fill the form that input will directly store in the excel sheet. Once everyone is finished I need to get that automatically filled excel sheet. Is it possible in python? Please can someone guide me on this how to proceed...
Yeah, try to show some code to prove you've tried first.
You could start will building the data structure you want for your application. Like
Name | Hours Spent for this week | Current Time
Then, you can use
import csv
to create a csv file to do so.
CSV module is impleted in python, there is also an excel one, but I highly recommend CSV, because it is easier to be processed by other programs.