I followed the tutorial on live graph plot from CSV/TXT file, but when I am running the python program no graph is created, instead, the terminal goes into busy mode until I exit using 'Ctrl+Z'.
For some reason, the animate function in matplotlib is not working for me. Instead I wrote the following code, which is supposed to do the job:
import matplotlib.pyplot as plt
while True:
pullData = open("data1.csv","r").read()
dataArray = pullData.split('\n')
xar = []
yar = []
for eachLine in dataArray:
if len(eachLine)>1:
x,y = eachLine.split(',')
xar.append(x)
yar.append(y)
plt.plot(xar, yar)
plt.pause(0.05)
plt.show()
But the above code is not reading the data points from the CSV file properly and generating the wrong graph.
I currently have Python 3.6.5 :: Anaconda, Inc. installed on the system. Could someone help in this, please? Thank you in advance.
You could use the polt Python package which I developed for this exact purpose of displaying live data.
Supposing you want to display live timeseries of multiple data columns in a CSV file, you could just pipe the live CSV stream (header+live columns) into polt:
(head -n1 myfile.csv; tail -fn0 myfile.csv) | polt add-source -p csv live
Explanation
(
head -n1 myfile.csv; # output first line of CSV file (header)
tail -fn0 myfile.csv # output new CSV data continuously
) | polt \ # pipe the data into polt
add-source -p csv # tell polt to interpret data as CSV
live # do the live plotting
If you do not directly want to plot timeseries, you can check the polt Animator documentation for further displaying possibilities.
Related
problem
I have a problem with matplotlib. I looked at a lot of things online (including an extensive amount of stack overflow questions) and did not find an answer to my problem. To be clear: I know how to dynamically update a plot in matplotlib. Everything I find online deals with showing the plot AFTER having set the series. here the problem is different. Please see the code below.
I have simulation results being written on the fly in a file (let's say file.log). I have a current code that prints the results on the fly as well using matplotlib. the problem is that, for various reasons, sometimes a new variable is written in file.log. But the monitoring is already displayed. How can I add a lineplot to the existing plot and keep updating ?
Here I give the general structure of the code I have.
The code structure I have
shell command:
I tail the log file one line by one line and pipe the output to a python script which handles the line in a for loop (this is working fine, no problem with that).
tail -f -n 1 file.log | monitoring.py
The python. It is commented line by line. Sorry for the long code sequence but everything with matplotlib is a bit long and needs context
#here I load the previous data (before starting monitoring)
historic_data = function_to_load_history('path/to/file.log')
#then I create a plot
figure, ax = plt.subplots()
#I keep the lines in a list
lineList = []
for serie in historic_data:
line, = ax.plot(serie)
lineList.append(line)
#then I display the plot
plt.ion()
plt.show()
#and I start 'listening' for tail input in the for loop
for line in sys.stdin:
# I add the new data to the historic data
historic_data = function_to_append(line, historic_data)
#I update the plot and redraw, using the already created lines
for i,serie in enumerate(historic_data):
# OK here is the problem : here, sometimes, a new serie is discovered in the log file
# so the last series have no line associated in lineList. I test this :
# if the line exist in the line list, then no problem I update an existing line
if i < len(lineList):
lineList[i].set_data(serie)
#HERE IS THE PROBLEM : if i enconter a new serie, I add it to the plot but it is not dislayed.
else:
line, = ax.plot(serie)
lineList.append(line)
figure.gca().autoscale_view()
figure.gca().relim()
plt.draw()
I have no idea how to ask matplotlib to draw the new serie(s). Any idea ?
thank you =)
I used the plotly streaming API from Python plot.ly/python/streaming-tutorial ) to set up a dashboard with graphs showing data streamed from local logfiles (.txt).
I followed the tutorial to create a graph of a data stream; reproducing the "Getting started" example worked fine (although i had to change the py.iplot() into py.plot()).
I made some small modifications to the working example code to get the Y-axis value from a text file on my local drive. It does manage to plot the value written in my text file on the graph and even update it as I modify the value in the text file, but it behaves differently than the graph produced by the example code for a streamed plotly graph. I include both my code for the "Example" graph and my code for the "Data from local text file" graph; and images of the different behaviors.
The first two images show the Plot and Data produced by the "Example" code and the last two for the "Data from local text file" code. : http://imgur.com/a/ugo6m
The interesting thing here is that in the first case (Example), the updated value of Y is shown on a new line in the Data tab. This is not the case in the second case (Data from local text file). In the second case, the Y value is updated, but always takes the place of the first line. Instead of adding a new Y point and storing the previous one, it just constantly modifies the first value that Y received. I think the problem comes from there.
Here's a link for both codes, they're short and only the last few lines matter, as I suppose the problem comes from there since they're the only difference between both codes. I tried different working expressions to read the value from the text file ("with open('data.txt', 'r')) but nothing does it. Does anyone know how to make it work properly?
(!!!Careful both codes run an infinite loop!!!)
"Example": http://pastebin.com/6by30ANs
"Data from local text file": see below
Thanks in advance for your time,
PS: I had to put my second code here below as I do not have enough reputation to put more than 2 links.
import plotly.plotly as py
import plotly.tools as tls
import plotly.graph_objs as go
import datetime
import time
tls.set_credentials_file(username='lo.dewaele', stream_ids = ['aqwoq4i2or'], api_key='PNASXMZQmLmAVLtthYq2')
stream_ids = tls.get_credentials_file()['stream_ids']
stream_id = stream_ids[0]
stream_1 = dict(token=stream_id, maxpoints=20)
trace1 = go.Scatter(
x=[],
y=[],
mode='lines+markers',
stream=stream_1
)
data = go.Data([trace1])
layout = go.Layout(title='Time Series')
fig = go.Figure(data=data, layout=layout)
py.plot(fig, filename='stream')
s = py.Stream(stream_id)
s.open()
time.sleep(1)
while True:
graphdata = open('graphdata.txt', 'r') #open file in read mode
y = [graphdata.readline()] #read the first line of the file (just one integer)
x = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')
s.write(dict(x=x, y=y))
graphdata.close() #close the file
time.sleep(1)
s.close()
I am working on developing a program in python that can display trending data from multiple csv files that are created each day. In each of those files are two columns of information (example):
Name | Downloads
Facebook 42
Myspace 3
Foursquare 12
The information is tracking a set of about 15 or so programs and how many times it was downloaded that particular day. I need to write a program that can take all those csv files and create a chart that shows growth or decline. The chart would have multiple colored lines with a legend displaying what each color represents. The chart would be segment by day and the lines would go across rising and falling depending on what the daily total was from the csv file. I'm just not sure where to start with the program.
Edit to include some work done. I'm using Jupyter to run and display the results. I would like to eventually incorporate Bokeh to easily change fields.
import -----
glob.glob(directory + '/' + "daily*.csv)
all_data = pd.DataFrame()
for f in glob.glob(directory + '/" + "daily*.csv):
df = pd.read_csv(f)
all_data = all_data.append(df,ignore_index=True)
status = pd.read_csv(f)
status
csv is displayed
records = pd.read_csv(f)
A dictionary is created here to group all the daily apps into all_files = []. After that I create a bar chart that displays in order all the files from most downloaded to least.
plt.gcf().subplots_adjust(bottom=0.05)
width = 30
scale = 55
fig, ax = plt.subplots()
plt.bar(range(0, scale*len(top_files[0]), scale), top_files[1], width)
ax.set_xticks([scale*i+width/2 for i in range(scale*len(top_files[0]))])
ax.set_xticklables([app for app in top_files[0]], rotation='vertical')
ax.xaxis.grid(False)
plt.ylabel("Total")
plt.ylim((0,250))
plt.xlim((0, scale*len(top_files[0])))
ax.set_aspect(6)
plt.tight_layout()
plt.show()
The only difficult part of all of this is the computer that all the code resides on is not, and will not be connected to the internet. So most work has to be repeated and copied, which is why I skipped some boring parts like dictionaries and imports.
This is also just one example of what I am working with. I've has written up a heat map to display a larger amount of data, but in the end I'm stumped on how to get multiple lines to display across a chart.
My data is look like as in the picture. All of my datas are in .txt format and my aim is to loop over files and plot them. First row represents my variables
(WL, ABS, T%) so firstly I need to delete them before proceeding.
with open('Desktop/100-3.txt', 'r') as f:
data = f.read().splitlines(True)
with open('Desktop/100-3.txt', 'w') as f:
f.writelines(data[1:])
Probably it would not be necessary but I am very new in Numpy. Basically the algorithm will be as follows:
Read all the .txt files
Plot T% versus WL, plot ABS versus WL, save. (WL -> x variable)
Continue for the next file, .. (two graphs for every .txt file)
Then finish the loop, exit.
data looks like this
What I've tried
from numpy import loadtxt
import os
dizin = os.listdir(os.getcwd())
for i in dizin:
if i.endswith('.txt'):
data = loadtxt("??",float)
For data files like this I would prefer np.genfromtxt over np.loadtxt, it has many useful options you can look up in the docs. The glob module is also nice to iterate over directories with wildcards as filters:
from glob import glob
import numpy as np
import matplotlib.pyplot as plt
# loop over all files in the current directory ending with .txt
for fname in glob("./*.txt"):
# read file, skip header (1 line) and unpack into 3 variables
WL, ABS, T = np.genfromtxt(fname, skip_header=1, unpack=True)
# first plot
plt.plot(WL, T)
plt.xlabel('WL')
plt.ylabel('T%')
plt.show()
plt.clf()
# second plot
plt.plot(ABS, T)
plt.xlabel('WL')
plt.ylabel('ABS')
plt.show()
plt.clf()
The next step would be to do some research on matplotlib to make the plots look better.
Please let me know if the code does not work, I'll try to fix it then.
EDIT: Added plt.clf() to clear the figure before creating a new one.
I have a stack of CT-scan images. After processing (one image from those stack) CT-scan image using Matlab, I saved XY coordinates for each different boundary region in different Excel sheets as follows:
I = imread('myCTscan.jpeg');
BW = im2bw(I);
[coords, labeledImg] = bwboundaries(BW, 4, 'holes');
sheet = 1;
for n=1:length(coords);
xlswrite('fig.xlsx',coords{n,1},sheet,'A1');
sheet = sheet+1;
end
The next step is then to import this set of coordinates and plot it into Abaqus CAE Sketch for finite element analysis.
I figure out that my workflow is something like this:
Import Excel workbook
For each sheet in workbook:
2.1. For each row: read both column to get xy coordinates (each row has two column, x and y coordinate)
2.2. Put each xy coordinates inside a list
2.3. From list, sketch using spline method
Repeat step 2 for other sheets within the workbook
I searched for a while and found something like this:
from abaqus import *
lines= open('fig.xlsx', 'r').readlines()
pointList= []
for line in lines:
pointList.append(eval('(%s)' %line.strip()))
s1= mdb.models['Model-1'].ConstrainedSketch(name='mySketch', sheetSize=500.0)
s1.Spline(points= pointList)
But this only read XY coordinates from only one sheet and I'm stuck at step 3 above. Thus my problem is that how to read these coordinates in different sheets using Abaqus/Python (Abaqus 6.14, Python 2.7) script?
I'm new to Python programming, I can read and understand the syntax but can't write very well (I'm still struggling on how to import Python module in Abaqus). Manually type each coordinates (like in Abaqus' modelAExample.py tutorial) is practically impossible since each of my CT-scan image can have 100++ of boundary regions and 10k++ points.
I'm using:
Windows 7 x64
Abaqus 6.14 (with built in Python 2.7)
Excel 2013
Matlab 2016a with Image Processing Toolbox
You are attempting to read excel files as comma separated files. CSV files by definition can not have more than one tab. Your read command is interpreting the file as a csv and not allowing you to iterate over the tabs in your file (though it begs the question how your file is opening properly in the first place as you are saving an xlsx and reading a csv).
There are numerous python libraries that will parse and process XLS/XLSX files.
Take a look at pyxl and use it to read your file in.
You would likely use something like
from openpyxl import Workbook
(some commands to open the workbook)
listofnames=wb.sheetnames
for k in listofnames:
ws=wb.worksheets(k)
and then input your remaining commands.