Say I have a dataframe df where df.index consists of datetime objects, e.g.
> df.index[0]
datetime.date(2014, 5, 5)
If I plot it Pandas nicely preserves the datetime type in the plot, which allows the user to change the time-series sampling as well formatting options of the plot:
# Plot the dataframe:
f = plt.figure(figsize=(8,8))
ax = f.add_subplot(1,1,1)
lines = df.plot(ax=ax)
# Choose the sampling rate in terms of dates:
ax.xaxis.set_major_locator(matplotlib.dates.WeekdayLocator(byweekday=(0,1,2,3,4,5,6),
interval=1))
# We can also re-sample the X axis numerically if we want (e.g. every 4 steps):
N = 4
ticks = ax.xaxis.get_ticklocs()
ticklabels = [l.get_text() for l in ax.xaxis.get_ticklabels()]
ax.xaxis.set_ticks(ticks[-1::-N][::-1])
ax.xaxis.set_ticklabels(ticklabels[-1::-N][::-1])
# Choose a date formatter using a date-friendly syntax:
ax.xaxis.set_major_formatter(matplotlib.dates.DateFormatter('%b\n%d'))
plt.show()
However, the above does not work for a boxplot (the tick labels for the x axis are rendered empty)
:
df2.boxplot(column='A', by='created_dt',ax=ax, sym="k.")
# same code as above ...
It looks like in the last example, Pandas converts the x-axis labels into string type, so the formatter and locators don't work anymore.
This post re-uses solutions from the following threads:
Accepted answer to Pandas timeseries plot setting x-axis major and minor ticks and labels
Accepted answer to Pandas: bar plot xtick frequency
Why? How can I use boxplot in a way that allows me to use matplotlib date locators and formatters?
No, actually even the line plot is not working correctly, if you have the year show up, you will notice the problem: instead of being 2000 in the following example, the xticks are in 1989.
In [49]:
df=pd.DataFrame({'Val': np.random.random(50)})
df.index=pd.date_range('2000-01-02', periods=50)
f = plt.figure()
ax = f.add_subplot(1,1,1)
lines = df.plot(ax=ax)
ax.xaxis.set_major_formatter(matplotlib.dates.DateFormatter('%y%b\n%d'))
print ax.get_xlim()
(10958.0, 11007.0)
In [50]:
matplotlib.dates.strpdate2num('%Y-%M-%d')('2000-01-02')
Out[50]:
730121.0006944444
In [51]:
matplotlib.dates.num2date(730121.0006944444)
Out[51]:
datetime.datetime(2000, 1, 2, 0, 1, tzinfo=<matplotlib.dates._UTC object at 0x051FA9F0>)
Turns out datetime data is handled differently in pandas and matplotlib: in the latter, 2000-1-2 should be 730121.0006944444, instead of 10958.0 in pandas
To get it right we need to avoid using pandas's plot method:
In [52]:
plt.plot_date(df.index.to_pydatetime(), df.Val, fmt='-')
ax=plt.gca()
ax.xaxis.set_major_formatter(matplotlib.dates.DateFormatter('%y%b\n%d'))
Similarly for barplot:
In [53]:
plt.bar(df.index.to_pydatetime(), df.Val, width=0.4)
ax=plt.gca()
ax.xaxis.set_major_formatter(matplotlib.dates.DateFormatter('%y%b\n%d'))
Related
I encounter an issue with Matplotlib.dates.DateFormatter :
I want to convert timestamps in Date format which is simple usually with the straftime but when using it on matplotlib i don't have the dynamic position on my graph so I used the md.DateFormatter('%H:%M:%S.%f') to have the X values as a date format with the dynamic index.
The fact is, my dates have too much values, I don't want the nanoseconds but I don't know how to remove them. I searched on StackOverflow to find a solution but applying a date[:-3] won't work as I have a datetime format...
Do you have a solution? It's maybe trivial but can't find any solution right now...
Thanks in advance.
NB : What I call the dynamic index is when you are on the graph and you can see the exact X and Y value of your pointer at the bottom
Here is an applicable example :
df =
timestamp val
0 2022-03-13 03:19:59.999070 X1
1 2022-03-13 03:20:00.004070 X2
2 2022-03-13 03:20:00.009070 X3
3 2022-03-13 03:20:00.014070 X4
And I try to plot this with :
ax=plt.gca()
xfmt = md.DateFormatter('%H:%M:%S.%f')
ax.xaxis.set_major_formatter(xfmt)
plt.plot(df.timestamp, df.val, linestyle="-", marker = ".")
plt.setp(ax.get_xticklabels(), rotation=40)
plt.show()
In conclusin, what I want is to remove the 070 in the graph but if I remove it beforehand, DateFormatter will replace it by 000 which is as useless as it was..
If you want to change both the tick labels and the format of the number shown on the interactive status bar, you could define your own function to deliver your desired format, then use a FuncFormatter to display those values on your plot.
For example:
import matplotlib.pyplot as plt
import matplotlib.dates as md
import pandas as pd
# dummy data
ts = pd.date_range("2022-03-13 03:19:59.999070",
"2022-03-13 03:20:00.014070", periods=4)
df = pd.DataFrame({'timestamp': ts, 'val':[0, 1, 2, 3]})
fig, ax = plt.subplots()
# define our own function to drop the last three characters
xfmt = lambda x, pos: md.DateFormatter('%H:%M:%S.%f')(x)[:-3]
# use that function as the major formatter, using FuncFormatter
ax.xaxis.set_major_formatter(plt.FuncFormatter(xfmt))
plt.setp(ax.get_xticklabels(), rotation=40)
ax.plot(df.timestamp, df.val, linestyle="-", marker = ".")
plt.tight_layout()
plt.show()
Note the matching tick format and status bar format.
If, however, you do not want to change the tick labels, but only change the value on the status bar, we can do that by reassigning the ax.format_coord function, using the a similar idea for the function we defined above, but also adding in the y value for display
For example:
import matplotlib.pyplot as plt
import matplotlib.dates as md
import pandas as pd
# dummy data
ts = pd.date_range("2022-03-13 03:19:59.999070",
"2022-03-13 03:20:00.014070", periods=4)
df = pd.DataFrame({'timestamp': ts, 'val':[0, 1, 2, 3]})
fig, ax = plt.subplots()
xfmt = md.DateFormatter('%H:%M:%S.%f')
xfmt2 = lambda x, y: "x={}, y={:g}".format(xfmt(x)[:-3], y)
# use original formatter here with microseconds
ax.xaxis.set_major_formatter(plt.FuncFormatter(xfmt))
# and the millisecond function here
ax.format_coord = xfmt2
plt.setp(ax.get_xticklabels(), rotation=40)
ax.plot(df.timestamp, df.val, linestyle="-", marker = ".")
plt.tight_layout()
plt.show()
Note the difference between the status bar and the tick formats here.
I wonder if it's possible to change the measurement milestones for graphs created by pandas. In my code the X-axis stands for time and is measured by month, but the measurement milestones are all over the place.
In the image below, the milestones for the X-axis are 2012M01, 2012M06, 2012M11, 2013M04 and 2013M09.
Is there any way I can choose how long the distance should be between every milestone? For example, to make it so it shows every year or every half year?
This is the code I used for the function making the graph:
def graph(dataframe):
graph = dataframe[["Profit"]].plot()
graph.set_title('Statistics')
graph.set_ylabel('Thousand $')
graph.set_xlabel('Time')
plt.grid(True)
plt.show()
The actual dataframe is just an excel-file with a bunch of months and monetary values in it.
I think the most straight forward is to use matplotlib.dates to format the axis:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
def graph(dataframe):
fig, ax = plt.subplots()
xfmt = mdates.DateFormatter('%YM%m') #see https://strftime.org/
major = mdates.MonthLocator([1,7]) #label only Jan and Jul
graph = dataframe[["Profit"]].plot(ax=ax) #link plot to the existing axes
graph.set_title('Statistics')
graph.set_ylabel('Thousand $')
graph.set_xlabel('Time')
graph.xaxis.set_major_locator(major) #set major locator tick on x-axis
graph.xaxis.set_major_formatter(xfmt) #format xtick label
plt.grid(True)
plt.show()
But a key point is you need to have your dates as Python's built-in datetime.date (not datetime.datetime); thanks to this answer. If your dates are str or a different type of datetime, you will need to convert, but there are many resources on SO and elsewhere for doing this like this or this:
In[0]:
dr = pd.date_range('01-01-2012', '01-01-2014', freq='1MS')
dr = [pd.to_datetime(date).date() for date in df.index] #explicitly converting to datetime with .date()
df = pd.DataFrame(index=dr, data={'Profit':np.random.rand(25)})
type(df.index.[0])
Out[0]:
datetime.date
Calling graph(df) using the example above gets this plot:
Just to expand on this, here's what happens when the index is pandas.Timestamp instead of datetime.date:
In[0]:
dr = pd.date_range('01-01-2012', '01-01-2014', freq='1MS')
# dr = [pd.to_datetime(date).date() for date in df.index] #skipping date conversion
df = pd.DataFrame(index=dr, data={'Profit':np.random.rand(25)})
graph(df)
Out[0]:
The x-axis is improperly formatted:
However, if you are willing to just create the plot directly through matplotlib, rather than pandas (pandas is using matplotlib anyway), this can handle more types of dates:
In[0]:
dr = pd.date_range('01-01-2012', '01-01-2014', freq='1MS')
# dr = [pd.to_datetime(date).date() for date in df.index] #skipping date conversion
df = pd.DataFrame(index=dr, data={'Profit':np.random.rand(25)})
def graph_2(dataframe):
fig, ax = plt.subplots()
xfmt = mdates.DateFormatter('%YM%m')
major = mdates.MonthLocator([1,7])
ax.plot(dataframe.index,dataframe['Profit'], label='Profit')
ax.set_title('Statistics')
ax.set_ylabel('Thousand $')
ax.set_xlabel('Time')
ax.xaxis.set_major_locator(major)
ax.xaxis.set_major_formatter(xfmt)
ax.legend() #legend needs to be added
plt.grid(True)
plt.show()
graph_2(df)
type(df.index[0])
Out[0]:
pandas._libs.tslibs.timestamps.Timestamp
And here is the working graph:
Following up my previous question: Sorting datetime objects by hour to a pandas dataframe then visualize to histogram
I need to plot 3 bars for one X-axis value representing viewer counts. Now they show those under one minute and above. I need one showing the overall viewers. I have the Dataframe but I can't seem to make them look right. With just 2 bars I have no problem, it looks just like I would want it with two bars:
The relevant part of the code for this:
# Time and date stamp variables
allviews = int(df['time'].dt.hour.count())
date = str(df['date'][0].date())
hours = df_hist_short.index.tolist()
hours[:] = [str(x) + ':00' for x in hours]
The hours variable that I use to represent the X-axis may be problematic, since I convert it to string so I can make the hours look like 23:00 instead of just the pandas index output 23 etc. I have seen examples where people add or subtract values from the X to change the bars position.
fig, ax = plt.subplots(figsize=(20, 5))
short_viewers = ax.bar(hours, df_hist_short['time'], width=-0.35, align='edge')
long_viewers = ax.bar(hours, df_hist_long['time'], width=0.35, align='edge')
Now I set the align='edge' and the two width values are absolutes and negatives. But I have no idea how to make it look right with 3 bars. I didn't find any positioning arguments for the bars. Also I have tried to work with the plt.hist() but I couldn't get the same output as with the plt.bar() function.
So as a result I wish to have a 3rd bar on the graph shown above on the left side, a bit wider than the other two.
pandas will do this alignment for you, if you make the bar plot in one step rather than two (or three). Consider this example (adapted from the docs to add a third bar for each animal).
import pandas as pd
import matplotlib.pyplot as plt
speed = [0.1, 17.5, 40, 48, 52, 69, 88]
lifespan = [2, 8, 70, 1.5, 25, 12, 28]
height = [1, 5, 20, 3, 30, 6, 10]
index = ['snail', 'pig', 'elephant',
'rabbit', 'giraffe', 'coyote', 'horse']
df = pd.DataFrame({'speed': speed,
'lifespan': lifespan,
'height': height}, index=index)
ax = df.plot.bar(rot=0)
plt.show()
In pure matplotlib, instead of using the width parameter to position the bars as you've done, you can adjust the x-values for your plot:
import numpy as np
import matplotlib.pyplot as plt
# Make some fake data:
n_series = 3
n_observations = 5
x = np.arange(n_observations)
data = np.random.random((n_observations,n_series))
# Plotting:
fig, ax = plt.subplots(figsize=(20,5))
# Determine bar widths
width_cluster = 0.7
width_bar = width_cluster/n_series
for n in range(n_series):
x_positions = x+(width_bar*n)-width_cluster/2
ax.bar(x_positions, data[:,n], width_bar, align='edge')
In your particular case, seaborn is probably a good option. You should (almost always) try keep your data in long-form so instead of three separate data frames for short, medium and long, it is much better practice to keep a single data frame and add a column that labels each row as short, medium or long. Use this new column as the hue parameter in Seaborn's barplot
I am plotting time series using pandas .plot() and want to see every month shown as an x-tick.
Here is the dataset structure
Here is the result of the .plot()
I was trying to use examples from other posts and matplotlib documentation and do something like
ax.xaxis.set_major_locator(
dates.MonthLocator(revenue_pivot.index, bymonthday=1,interval=1))
But that removed all the ticks :(
I also tried to pass xticks = df.index, but it has not changed anything.
What would be the rigth way to show more ticks on x-axis?
No need to pass any args to MonthLocator. Make sure to use x_compat in the df.plot() call per #Rotkiv's answer.
import pandas as pd
import numpy as np
import matplotlib.pylab as plt
import matplotlib.dates as mdates
df = pd.DataFrame(np.random.rand(100,2), index=pd.date_range('1-1-2018', periods=100))
ax = df.plot(x_compat=True)
ax.xaxis.set_major_locator(mdates.MonthLocator())
plt.show()
formatted x-axis with set_major_locator
unformatted x-axis
You could also format the x-axis ticks and labels of a pandas DateTimeIndex "manually" using the attributes of a pandas Timestamp object.
I found that much easier than using locators from matplotlib.dates which work on other datetime formats than pandas (if I am not mistaken) and thus sometimes show an odd behaviour if dates are not converted accordingly.
Here's a generic example that shows the first day of each month as a label based on attributes of pandas Timestamp objects:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# data
dim = 8760
idx = pd.date_range('1/1/2000 00:00:00', freq='h', periods=dim)
df = pd.DataFrame(np.random.randn(dim, 2), index=idx)
# select tick positions based on timestamp attribute logic. see:
# https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Timestamp.html
positions = [p for p in df.index
if p.hour == 0
and p.is_month_start
and p.month in range(1, 13, 1)]
# for date formatting, see:
# https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior
labels = [l.strftime('%m-%d') for l in positions]
# plot with adjusted labels
ax = df.plot(kind='line', grid=True)
ax.set_xlabel('Time (h)')
ax.set_ylabel('Foo (Bar)')
ax.set_xticks(positions)
ax.set_xticklabels(labels)
plt.show()
yields:
Hope this helps!
The right way to do that described here
Using the x_compat parameter, it is possible to suppress automatic tick resolution adjustment
df.A.plot(x_compat=True)
If you want to just show more ticks, you can also dive deep into the structure of pd.plotting._converter:
dai = ax.xaxis.minor.formatter.plot_obj.date_axis_info
dai['fmt'][dai['fmt'] == b''] = b'%b'
After plotting, the formatter is a TimeSeries_DateFormatter and _set_default_format has been called, so self.plot_obj.date_axis_info is not None. You can now manipulate the structured array .date_axis_info to be to your liking, namely contain less b'' and more b'%b'
Remove tick labels:
ax = df.plot(x='date', y=['count'])
every_nth = 10
for n, label in enumerate(ax.xaxis.get_ticklabels()):
if n % every_nth != 0:
label.set_visible(False)
Lower every_nth to include more labels, raise to keep fewer.
I have the foll. dataframe:
Av_Temp Tot_Precip
278.001 0
274 0.0751864
270.294 0.631634
271.526 0.229285
272.246 0.0652201
273 0.0840059
270.463 0.0602944
269.983 0.103563
268.774 0.0694555
269.529 0.010908
270.062 0.043915
271.982 0.0295718
and want to plot a boxplot where the x-axis is 'Av_Temp' divided into equi-sized bins (say 2 in this case), and the Y-axis shows the corresponding range of values for Tot_Precip. I have the foll. code (thanks to Find pandas quartiles based on another column), however, when I plot the boxplots, they are getting plotted one on top of another. Any suggestions?
expl_var = 'Av_Temp'
cname = 'Tot_Precip'
df[expl_var+'_Deciles'] = pandas.qcut(df[expl_var], 2)
grp_df = df.groupby(expl_var+'_Deciles').apply(lambda x: numpy.array(x[cname]))
fig, ax = plt.subplots()
for i in range(len(grp_df)):
box_arr = grp_df[i]
box_arr = box_arr[~numpy.isnan(box_arr)]
stats = cbook.boxplot_stats(box_arr, labels = str(i))
ax.bxp(stats)
ax.set_yscale('log')
plt.show()
Since you're using pandas already, why not use the boxplot method on dataframes?
expl_var = 'Av_Temp'
cname = 'Tot_Precip'
df[expl_var+'_Deciles'] = pandas.qcut(df[expl_var], 2)
ax = df.boxplot(by='Av_Temp_Deciles', column='Tot_Precip')
ax.set_yscale('log')
That produces this: http://i.stack.imgur.com/20KPx.png
If you don't like the labels, throw in a
plt.xlabel('');plt.suptitle('');plt.title('')
If you want a standard boxplot, the above should be fine. My understanding of the separation of boxplot into boxplot_stats and bxp is to allow you to modify or replace the stats generated and fed to the plotting routine. See https://github.com/matplotlib/matplotlib/pull/2643 for some details.
If you need to draw a boxplot with non-standard stats, you can use boxplot_stats on 2D numpy arrays, so you only need to call it once. No loops required.
expl_var = 'Av_Temp'
cname = 'Tot_Precip'
df[expl_var+'_Deciles'] = pandas.qcut(df[expl_var], 2)
# I moved your nan check into the df apply function
grp_df = df.groupby('Av_Temp_Deciles').apply(lambda x: numpy.array(x[cname][~numpy.isnan(x[cname])]))
# boxplot_stats can take a 2D numpy array of data, and a 1D array of labels
# stats is now a list of dictionaries of stats, one dictionary per quantile
stats = cbook.boxplot_stats(grp_df.values, labels=grp_df.index)
# now it's a one-shot plot, no loops
fig, ax = plt.subplots()
ax.bxp(stats)
ax.set_yscale('log')