I'm trying to plot time series data using matplotlib using the following code
import matplotlib.pyplot as plt
x, y = data.shape
y_metrics = alarms_metrics + udp_alarms_metrics_names
print x, y
for j in range(1, y):
time_stamp = []
metric = []
plt.figure()
for i in range(x):
time_stamp.append(data_time[i][0])
metric.append(data[i][j])
plt.xlabel("Time")
plt.ylabel(str(y_metrics[j-1]))
plt.plot(time_stamp, metric)
plt.savefig(str(j))
The format of time_stamp is
['2017:02:01:14:44:00', '2017:02:01:14:44:01',...etc]
time_stamp is a list of type 'numpy.string_'
I'm getting the following error
Traceback (most recent call last):
File "/home/joseph/PycharmProjects/demo/src/alarm_log/alarm_db_plot.py", line 23, in <module>
plt.plot(time_stamp, metric)
File "/usr/lib/python2.7/dist-packages/matplotlib/pyplot.py", line 3154, in plot
ret = ax.plot(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/matplotlib/__init__.py", line 1814, in inner
return func(ax, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/matplotlib/axes/_axes.py", line 1425, in plot
self.add_line(line)
File "/usr/lib/python2.7/dist-packages/matplotlib/axes/_base.py", line 1708, in add_line
self._update_line_limits(line)
File "/usr/lib/python2.7/dist-packages/matplotlib/axes/_base.py", line 1730, in _update_line_limits
path = line.get_path()
File "/usr/lib/python2.7/dist-packages/matplotlib/lines.py", line 925, in get_path
self.recache()
File "/usr/lib/python2.7/dist-packages/matplotlib/lines.py", line 612, in recache
x = np.asarray(xconv, np.float_)
File "/usr/local/lib/python2.7/dist-packages/numpy/core/numeric.py", line 482, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: invalid literal for float(): 2017:02:01:14:44:00
I think I might need to used matplotlib dateformatter to parse the time_stamp. I tried so but I couldn't figure out how. Any help please?
Not exactly sure what you're trying to plot without seeing the data, but you could try to convert the string to a datetime object. See here for a list with the format codes.
from datetime import datetime
time_stamp = ['2017:02:01:14:44:00', '2017:02:01:14:44:01']
time_stamp = [datetime.strptime(i, '%Y:%m:%d:%H:%M:%S') for i in time_stamp]
print(time_stamp)
Related
I have a timesheet.csv that I am trying to plot as a gantt chart.
Not quite sure what the issue is as my understanding is that the start_num is where the hbar begins and the end_num is where the hbar is supposed to end.
import pandas as pd
import matplotlib.pyplot as plt
f = 'timesheet_example.csv'
df = pd.read_csv(f)
df['total_hours'] = pd.to_datetime(df['total_hours'], format='%H')
df['start_time'] = pd.to_datetime(df['start_time'])
df['end_time'] = pd.to_datetime(df['end_time'])
# period start date
period_start = df.start_time.min()
# number of days from period_start to start of tasks
df['start_num'] = (df.start_time - period_start)
# number of days from period_start to end of tasks
df['end_num'] = (df.end_time - period_start)
# days between start and end of each taks
df['num_start_to_end'] = (df.end_num - df.start_num)
fig, ax = plt.subplots(1, figsize=(16,6))
ax.barh(df.client_name, df.num_start_to_end, left=df.start_num)
plt.show()
Ideally, the gantt plot would be an approximate work day (9-5) with the intentions of making it either 15min or 1hr block ticks.
Console error:
File "P:\xxxxxxxxxxxxx\Timesheet.py", line 38, in <module>
ax.barh(df.client_name, df.num_start_to_end, left=df.start_num)
File "C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\axes\_axes.py", line 2551, in barh
patches = self.bar(x=left, height=height, width=width, bottom=y,
File "C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\__init__.py", line 1412, in inner
return func(ax, *map(sanitize_sequence, args), **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\axes\_axes.py", line 2409, in bar
self.add_patch(r)
File "C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\axes\_base.py", line 2363, in add_patch
self._update_patch_limits(p)
File "C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\axes\_base.py", line 2386, in _update_patch_limits
patch_trf = patch.get_transform()
File "C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\patches.py", line 278, in get_transform
return self.get_patch_transform() + artist.Artist.get_transform(self)
File "C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\patches.py", line 758, in get_patch_transform
bbox = self.get_bbox()
File "C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\patches.py", line 851, in get_bbox
return transforms.Bbox.from_extents(x0, y0, x1, y1)
File "C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\transforms.py", line 839, in from_extents
bbox = Bbox(np.reshape(args, (2, 2)))
File "C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\transforms.py", line 775, in __init__
points = np.asarray(points, float)
TypeError: float() argument must be a string or a number, not 'Timedelta'
I faced with the problem during telegram bot writing. I would be very happy if somebody help me with this.
My code
import telebot
import matplotlib.pyplot as plt
import numpy as np
...
def plot_func(message):
x = np.linspace(-5,5,100)
y = message.text # <-- here is something wrong I supppose
plt.plot(x, y, 'r')
plt.savefig('plot_name.png', dpi = 300)
bot.send_photo(message.chat.id, photo=open('plot_name.png', 'rb'))
#plt.show()
Main idea
User send function, bot plot it and send image back:
ERROR
/home/anmnv/Desktop/news_scrapper_bot/bot.py:148: UserWarning: Starting a Matplotlib GUI outside of the main thread will likely fail.
plt.plot(x, y, 'r')
Traceback (most recent call last):
File "/home/anmnv/Desktop/news_scrapper_bot/bot.py", line 424, in <module>
bot.polling(none_stop=True)
File "/home/anmnv/.local/lib/python3.10/site-packages/telebot/__init__.py", line 1047, in polling
self.__threaded_polling(non_stop=non_stop, interval=interval, timeout=timeout, long_polling_timeout=long_polling_timeout,
File "/home/anmnv/.local/lib/python3.10/site-packages/telebot/__init__.py", line 1122, in __threaded_polling
raise e
File "/home/anmnv/.local/lib/python3.10/site-packages/telebot/__init__.py", line 1078, in __threaded_polling
self.worker_pool.raise_exceptions()
File "/home/anmnv/.local/lib/python3.10/site-packages/telebot/util.py", line 154, in raise_exceptions
raise self.exception_info
File "/home/anmnv/.local/lib/python3.10/site-packages/telebot/util.py", line 98, in run
task(*args, **kwargs)
File "/home/anmnv/Desktop/news_scrapper_bot/bot.py", line 148, in plot_func
plt.plot(x, y, 'r')
File "/home/anmnv/.local/lib/python3.10/site-packages/matplotlib/pyplot.py", line 2730, in plot
return gca().plot(
File "/home/anmnv/.local/lib/python3.10/site-packages/matplotlib/axes/_axes.py", line 1662, in plot
lines = [*self._get_lines(*args, data=data, **kwargs)]
File "/home/anmnv/.local/lib/python3.10/site-packages/matplotlib/axes/_base.py", line 311, in __call__
yield from self._plot_args(
File "/home/anmnv/.local/lib/python3.10/site-packages/matplotlib/axes/_base.py", line 504, in _plot_args
raise ValueError(f"x and y must have same first dimension, but "
ValueError: x and y must have same first dimension, but have shapes (100,) and (1,)
Thank you in advance
Assuming message.text contains the string 'x**2', you can use numexpr.evaluate to convert to numpy array:
import numexpr
import matplotlib.pyplot as plt
x = np.linspace(-5, 5, 100)
y = numexpr.evaluate(message.text) # message.text = 'x**2'
plt.plot(x, y, 'r')
Output:
I'm reading TESS data and as you might expect it can be really large, My Buffer is too small. Is there a way I can avoid this error? Like maybe skip the file that has a large file? Or is there a more permanent solution(not involving more memory)? My Code is below, along with the Full Error
from lightkurve import TessTargetPixelFile
import lightkurve as lk
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
def func(tpf):
tpf.plot(aperture_mask=tpf.pipeline_mask);
lc = tpf.to_lightcurve()
mask = (lc.time.value < 1464)
masked_lc = lc[mask]
clipped_lc = masked_lc.remove_outliers(sigma=5);
flat_lc = clipped_lc.flatten()
binned_lc = flat_lc.bin(binsize=5)
periodogram = binned_lc.to_periodogram(method="bls", period=np.arange(1, 20, 0.001))
print(periodogram.plot())
planet_b_period = periodogram.period_at_max_power
planet_b_t0 = periodogram.transit_time_at_max_power
planet_b_dur = periodogram.duration_at_max_power
ax = binned_lc.fold(period=planet_b_period, epoch_time=planet_b_t0).scatter()
ax.set_xlim(-5, 5);
best_fit_period = periodogram.period_at_max_power
print('Best fit period: {:.3f}'.format(best_fit_period))
bfc=best_fit_period
print(bfc)
folded_lc = binned_lc.fold(period=bfc)
folded_lc.scatter(s=7);
plt.show()
planet_b_period
planet_b_model = periodogram.get_transit_model(period=planet_b_period,
transit_time=planet_b_t0,
duration=planet_b_dur)
ax = binned_lc.fold(planet_b_period, planet_b_t0).scatter(s=7)
planet_b_model.fold(planet_b_period, planet_b_t0).plot(ax=ax, c='r', lw=2)
ax.set_xlim(-5, 5);
dataset=pd.read_csv("all_targets_S001_v1.csv")
d=dataset["TICID"]
print(d)
IDs=[]
for i in range(len(d)):
IDs.append("TIC"+str(d[i]))
for i in range(len(IDs)):
tpf_file = lk.search_targetpixelfile(IDs[i], mission="TESS", sector=5).download(quality_bitmask='default')
try:
func(tpf_file)
continue
except:
continue
Thanks
The Full Error
WARNING: File may have been truncated: actual file length (262144) is smaller than the expected size (46402560) [astropy.io.fits.file]
Traceback (most recent call last):
File "lc.py", line 42, in <module>
tpf_file = lk.search_targetpixelfile(IDs[i], mission="TESS", sector=5).download(quality_bitmask='default')
File "C:\ProgramData\Anaconda3\lib\site-packages\lightkurve\utils.py", line 555, in wrapper
return f(*args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\lightkurve\search.py", line 355, in download
return self._download_one(
File "C:\ProgramData\Anaconda3\lib\site-packages\lightkurve\search.py", line 290, in _download_one
return read(path, quality_bitmask=quality_bitmask, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\lightkurve\io\read.py", line 112, in read
return getattr(__import__("lightkurve"), filetype)(path_or_url, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\lightkurve\targetpixelfile.py", line 2727, in __init__
quality_array=self.hdu[1].data["QUALITY"], bitmask=quality_bitmask
File "C:\ProgramData\Anaconda3\lib\site-packages\astropy\utils\decorators.py", line 758, in __get__
val = self.fget(obj)
File "C:\ProgramData\Anaconda3\lib\site-packages\astropy\io\fits\hdu\table.py", line 399, in data
data = self._get_tbdata()
File "C:\ProgramData\Anaconda3\lib\site-packages\astropy\io\fits\hdu\table.py", line 171, in _get_tbdata
raw_data = self._get_raw_data(self._nrows, columns.dtype,
File "C:\ProgramData\Anaconda3\lib\site-packages\astropy\io\fits\hdu\base.py", line 520, in _get_raw_data
return self._file.readarray(offset=offset, dtype=code, shape=shape)
File "C:\ProgramData\Anaconda3\lib\site-packages\astropy\io\fits\file.py", line 330, in readarray
return np.ndarray(shape=shape, dtype=dtype, offset=offset,
TypeError: buffer is too small for requested array
I'm trying to plot a pandas series, but I'm encountering an error when I attempt to format the x-axis date.
(A related issue was identified in the comments, but it appears that it was resolved in a much older version of pandas than what I'm using. So, it seems like this is a new problem.)
Consider a plot of the following pandas series:
import pandas as pd
d = {pd.Timestamp('2021-03-15 08:30:00'): -65.926651,
pd.Timestamp('2021-03-15 08:30:05'): -42.115551,
pd.Timestamp('2021-03-15 08:30:10'): -24.699627,
pd.Timestamp('2021-03-15 08:30:15'): -12.010081,
pd.Timestamp('2021-03-15 08:30:20'): -2.781321}
s = pd.Series(d)
ax = s.plot()
I seek to format the x-axis date on the plot using:
from matplotlib.dates import DateFormatter
format_str: str = '%H:%M:%S'
format_: DateFormatter = DateFormatter(format_str)
ax.xaxis.set_major_formatter(format_)
This results in the following error:
Traceback (most recent call last):
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/backends/backend_macosx.py", line 61, in _draw
self.figure.draw(renderer)
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
return draw(artist, renderer, *args, **kwargs)
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/figure.py", line 1863, in draw
mimage._draw_list_compositing_images(
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/image.py", line 131, in _draw_list_compositing_images
a.draw(renderer)
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
return draw(artist, renderer, *args, **kwargs)
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/cbook/deprecation.py", line 411, in wrapper
return func(*inner_args, **inner_kwargs)
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/axes/_base.py", line 2747, in draw
mimage._draw_list_compositing_images(renderer, self, artists)
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/image.py", line 131, in _draw_list_compositing_images
a.draw(renderer)
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
return draw(artist, renderer, *args, **kwargs)
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/axis.py", line 1164, in draw
ticks_to_draw = self._update_ticks()
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/axis.py", line 1022, in _update_ticks
major_labels = self.major.formatter.format_ticks(major_locs)
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/ticker.py", line 250, in format_ticks
return [self(value, i) for i, value in enumerate(values)]
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/ticker.py", line 250, in <listcomp>
return [self(value, i) for i, value in enumerate(values)]
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/dates.py", line 605, in __call__
return num2date(x, self.tz).strftime(self.fmt)
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/dates.py", line 511, in num2date
return _from_ordinalf_np_vectorized(x, tz).tolist()
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/numpy/lib/function_base.py", line 2108, in __call__
return self._vectorize_call(func=func, args=vargs)
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/numpy/lib/function_base.py", line 2192, in _vectorize_call
outputs = ufunc(*inputs)
File "/Users/me/VirtualEnvironments/my_venv/lib/python3.9/site-packages/matplotlib/dates.py", line 331, in _from_ordinalf
np.timedelta64(int(np.round(x * MUSECONDS_PER_DAY)), 'us'))
OverflowError: int too big to convert
Interestingly, if I add a fractional offset to the timestamps, everything works:
s.index += pd.DateOffset(seconds=0.5)
When I examine x in the np.timedelta64 call, it corresponds to the number of days since the start of the unix epoch (1 Jan 1970) only if I add a fractional part to the timestamp. If there's no fractional part, the resulting integer is huge and seems to have no obvious relationship to the number of days since 1 Jan 1970.
What's wrong here?
Error occurred because data was given that exceeded the number range handled by DateFormatter.
Please refer to the official reference.
For example, the actual data for the first time series looks like this
s.index[0].value
1615797000000000000
This needs to be converted to numbers that can be handled by matplotlib.
s.index = mdates.date2num(s.index)
s
18701.354167 -65.926651
18701.354225 -42.115551
18701.354282 -24.699627
18701.354340 -12.010081
18701.354398 -2.781321
dtype: float64
update(I am on 3.6.3, so I am fixing it.)
ax = s.plot(style='o-')
import matplotlib.dates as mdates
format_str = '%H:%M:%S'
format_ = mdates.DateFormatter(format_str)
ax.xaxis.set_major_formatter(format_)
I had the same error message and I used this to resolve it.
import matplotlib.dates as mdates
ts = mdates.epoch2num(ts)
After that, I didn't have any error. I hope it will be help you.
Following this answer's use of DateFormatter, I tried to plot a time series and label its x axis with years using pandas 0.15.0 and matplotlib 1.4.2:
import datetime as dt
import matplotlib as mpl
import matplotlib.pyplot as plt
import pandas.io.data as pdio
import scipy as sp
t1 = dt.datetime(1960, 1, 1)
t2 = dt.datetime(2014, 6, 1)
data = pdio.DataReader("GS10", "fred", t1, t2).resample("Q", how=sp.mean)
fig, ax1 = plt.subplots()
ax1.plot(data.index, data.GS10)
ax1.set_xlabel("Year")
ax1.set_ylabel("Rate (%)")
ax1.xaxis.set_major_formatter(mpl.dates.DateFormatter("%Y"))
fig.suptitle("10-yr Treasury Rate", fontsize=14)
fig.savefig('test.eps')
The final line throws an error: OverflowError: Python int too large to convert to C long
with this traceback:
C:\Anaconda3\lib\site-packages\IPython\core\formatters.py:239:
FormatterWarning: Exception in image/png formatter: Python int too
large to convert to C long FormatterWarning,
Traceback (most recent call
last):
File "", line 1, in
runfile('D:/username/latex_template/new_pandas_example.py', wdir='D:/username/latex_template')
File
"C:\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py",
line 580, in runfile
execfile(filename, namespace)
File
"C:\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py",
line 48, in execfile
exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)
File "D:/username/latex_template/new_pandas_example.py", line 18, in
fig.savefig('test.eps')
File "C:\Anaconda3\lib\site-packages\matplotlib\figure.py", line
1470, in savefig
self.canvas.print_figure(*args, **kwargs)
File "C:\Anaconda3\lib\site-packages\matplotlib\backend_bases.py",
line 2194, in print_figure
**kwargs)
File
"C:\Anaconda3\lib\site-packages\matplotlib\backends\backend_ps.py",
line 992, in print_eps
return self._print_ps(outfile, 'eps', *args, **kwargs)
File
"C:\Anaconda3\lib\site-packages\matplotlib\backends\backend_ps.py",
line 1020, in _print_ps
**kwargs)
File
"C:\Anaconda3\lib\site-packages\matplotlib\backends\backend_ps.py",
line 1110, in _print_figure
self.figure.draw(renderer)
File "C:\Anaconda3\lib\site-packages\matplotlib\artist.py", line 59,
in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "C:\Anaconda3\lib\site-packages\matplotlib\figure.py", line
1079, in draw
func(*args)
File "C:\Anaconda3\lib\site-packages\matplotlib\artist.py", line 59,
in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "C:\Anaconda3\lib\site-packages\matplotlib\axes_base.py", line
2092, in draw
a.draw(renderer)
File "C:\Anaconda3\lib\site-packages\matplotlib\artist.py", line 59,
in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "C:\Anaconda3\lib\site-packages\matplotlib\axis.py", line 1114,
in draw
ticks_to_draw = self._update_ticks(renderer)
File "C:\Anaconda3\lib\site-packages\matplotlib\axis.py", line 957,
in _update_ticks
tick_tups = [t for t in self.iter_ticks()]
File "C:\Anaconda3\lib\site-packages\matplotlib\axis.py", line 957,
in
tick_tups = [t for t in self.iter_ticks()]
File "C:\Anaconda3\lib\site-packages\matplotlib\axis.py", line 905,
in iter_ticks
for i, val in enumerate(majorLocs)]
File "C:\Anaconda3\lib\site-packages\matplotlib\axis.py", line 905,
in
for i, val in enumerate(majorLocs)]
File "C:\Anaconda3\lib\site-packages\matplotlib\dates.py", line 411,
in call
dt = num2date(x, self.tz)
File "C:\Anaconda3\lib\site-packages\matplotlib\dates.py", line 345,
in num2date
return _from_ordinalf(x, tz)
File "C:\Anaconda3\lib\site-packages\matplotlib\dates.py", line 225,
in _from_ordinalf
dt = datetime.datetime.fromordinal(ix)
OverflowError: Python int too large to convert to C long
Am I using DateFormatter incorrectly here? How can I easily put years (or any time format, since my time series might differ) on the a-axis of a matplotlib figure?
This is a 'regression' in pandas 0.15 (due to the refactor of Index), see https://github.com/matplotlib/matplotlib/issues/3727 and https://github.com/pydata/pandas/issues/8614, but is fixed in 0.15.1.
Short story: matplotlib now sees the pandas index as an array of datetime64[ns] values (which are actually very large int64s), instead of an array of Timestamps (which are subclass of datetime.datetime, and can be handled by matplotlib) in previous versions of pandas. So the underlying reason is that matplotlib does not handle datetime64 as date values but as ints.
For pandas 0.15.0 (but better upgrade to a newer version), there are two possible workarounds:
Register the datetime64 type, so it will also be handled as a date by matplotlib:
units.registry[np.datetime64] = pd.tseries.converter.DatetimeConverter()
Or convert the DatetimeIndex (with datetime64 values) to an array of datetime.datetime values with the to_pydatetime method, and plot this:
ax1.plot(data.index.to_pydatetime(), data.GS10)
related question: Plotting datetimeindex on x-axis with matplotlib creates wrong ticks in pandas 0.15 in contrast to 0.14