I would like to plot a bar graph that has only a few entries of data in each column of a pandas DataFrame with a bar graph. This is successful, but not only does it have the wrong y-axis limits, it also makes the x ticks very closely spaced so that the graph is useless. I would like to change the step rate to be about every week or so and only display day, month and year. I have the following DataFrame:
Observed WRF
2014-06-28 12:00:00 0.0 0.0
2014-06-28 13:00:00 0.0 0.0
2014-06-28 14:00:00 0.0 0.0
2014-06-28 15:00:00 0.0 0.0
2014-06-28 16:00:00 0.0 0.0
2014-06-28 17:00:00 0.0 0.0
2014-06-28 18:00:00 0.0 0.0
2014-06-28 19:00:00 0.0 0.0
2014-06-28 20:00:00 0.0 0.0
2014-06-28 21:00:00 0.0 0.0
2014-06-28 22:00:00 0.0 0.0
2014-06-28 23:00:00 0.0 0.0
2014-06-29 00:00:00 0.0 0.0
2014-06-29 01:00:00 0.0 0.0
2014-06-29 02:00:00 0.0 0.0
2014-06-29 03:00:00 0.0 0.0
2014-06-29 04:00:00 0.0 0.0
2014-06-29 05:00:00 0.0 0.0
2014-06-29 06:00:00 0.0 0.0
2014-06-29 07:00:00 0.0 0.0
2014-06-29 08:00:00 0.0 0.0
2014-06-29 09:00:00 0.0 0.0
2014-06-29 10:00:00 0.0 0.0
2014-06-29 11:00:00 0.0 0.0
2014-06-29 12:00:00 0.0 0.0
2014-06-29 13:00:00 0.0 0.0
2014-06-29 14:00:00 0.0 0.0
2014-06-29 15:00:00 0.0 0.0
2014-06-29 16:00:00 0.0 0.0
2014-06-29 17:00:00 0.0 0.0
... ...
2014-07-04 02:00:00 0.0002 0.0
2014-07-04 03:00:00 0.2466 0.0
2014-07-04 04:00:00 0.7103 0.0
2014-07-04 05:00:00 0.9158 1.93521e-13
2014-07-04 06:00:00 0.6583 0.0
2014-07-04 07:00:00 0.3915 0.0
2014-07-04 08:00:00 0.1249 0.0
2014-07-04 09:00:00 0.0 0.0
... ...
2014-08-30 07:00:00 0.0 0.0
2014-08-30 08:00:00 0.0 0.0
2014-08-30 09:00:00 0.0 0.0
2014-08-30 10:00:00 0.0 0.0
2014-08-30 11:00:00 0.0 0.0
2014-08-30 12:00:00 0.0 0.0
2014-08-30 13:00:00 0.0 0.0
2014-08-30 14:00:00 0.0 0.0
2014-08-30 15:00:00 0.0 0.0
2014-08-30 16:00:00 0.0 0.0
2014-08-30 17:00:00 0.0 0.0
2014-08-30 18:00:00 0.0 0.0
2014-08-30 19:00:00 0.0 0.0
2014-08-30 20:00:00 0.0 0.0
2014-08-30 21:00:00 0.0 0.0
2014-08-30 22:00:00 0.0 0.0
2014-08-30 23:00:00 0.0 0.0
2014-08-31 00:00:00 0.0 0.0
2014-08-31 01:00:00 0.0 0.0
2014-08-31 02:00:00 0.0 0.0
2014-08-31 03:00:00 0.0 0.0
2014-08-31 04:00:00 0.0 0.0
2014-08-31 05:00:00 0.0 0.0
2014-08-31 06:00:00 0.0 0.0
2014-08-31 07:00:00 0.0 0.0
2014-08-31 08:00:00 0.0 0.0
2014-08-31 09:00:00 0.0 0.0
2014-08-31 10:00:00 0.0 0.0
2014-08-31 11:00:00 0.0 0.0
2014-08-31 12:00:00 0.0 0.0
And the following code to plot it:
df4.plot(kind='bar',edgecolor='none',figsize=(16,8),linewidth=2, color=((1,0.502,0),'black'))
plt.legend(prop={'size':16})
plt.subplots_adjust(left=.1, right=0.9, top=0.9, bottom=.1)
plt.title('Five Day WRF Model Comparison Near %.2f,%.2f' %(lat,lon),fontsize=24)
plt.ylabel('Hourly Accumulated Precipitation [mm]',fontsize=18,color='black')
ax4=plt.gca()
maxs4=df4.max()
ax4.set_ylim([0, maxs4.max()])
ax4.xaxis_date()
ax4.xaxis.set_label_coords(0.5, -0.05)
plt.xlabel('Time',fontsize=18,color='black')
plt.show()
The y-axis starts at 0, but continues to about double the maximum value of the y-limit. The x-axis counts by hours, which is what I separated the data by, so that makes sense. However, it is not a helpful display.
Look at this code:
import pandas as pd
import numpy as np
from datetime import datetime
import matplotlib.pylab as plt
from matplotlib.dates import DateFormatter
# Sample data
df_origin = pd.DataFrame(pd.date_range(datetime(2014,6,28,12,0,0),
datetime(2014,8,30,12,0,0), freq='1H'), columns=['Valid Time'])
df_origin = df_origin .set_index('Valid Time')
df_origin ['Precipitation'] = np.random.uniform(low=0., high=10., size=(len(df_origin.index)))
df_origin .loc[20:100, 'Precipitation'] = 0.
df_origin .loc[168:168*2, 'Precipitation'] = 0. # second week has to be dry
# Plotting
df_origin.plot(y='Precipitation',kind='bar',edgecolor='none',figsize=(16,8),linewidth=2, color=((1,0.502,0)))
plt.legend(prop={'size':16})
plt.subplots_adjust(left=.1, right=0.9, top=0.9, bottom=.1)
plt.title('Precipitation (WRF Model)',fontsize=24)
plt.ylabel('Hourly Accumulated Precipitation [mm]',fontsize=18,color='black')
ax = plt.gca()
plt.gcf().autofmt_xdate()
# skip ticks for X axis
ax.set_xticklabels([dt.strftime('%Y-%m-%d') for dt in df_origin.index])
for i, tick in enumerate(ax.xaxis.get_major_ticks()):
if (i % (24*7) != 0): # 24 hours * 7 days = 1 week
tick.set_visible(False)
plt.xlabel('Time',fontsize=18,color='black')
plt.show()
Related
I have a dataframe which I pivoted and I now want to select spefici rows from the data. I have seen similar questions such as the one here: Selecting columns in a pandas pivot table based on specific row value?. In my case I want to return all the columns but I want to select only specific rows.
timestamp,value
2008-03-01 00:00:00,55.0
2008-03-01 00:15:00,20.0
2008-03-01 00:30:00,13.0
2008-03-01 00:45:00,78.0
2008-03-01 01:00:00,34.0
2008-03-01 01:15:00,123.0
2008-03-01 01:30:00,25.0
2008-03-01 01:45:00,91.0
2008-03-02 00:00:00,55.0
2008-03-02 00:15:00,46.0
2008-03-02 00:30:00,66.0
2008-03-02 00:45:00,24.0
2008-03-02 01:00:00,70.0
2008-03-02 01:15:00,32.0
2008-03-02 01:30:00,15.0
2008-03-02 01:45:00,92.0
I have done the below to generate the below output
import pandas as pd
import numpy as np
from datetime import datetime
df = pd.read_csv('df.csv')
df.timestamp = pd.to_datetime(df.timestamp)
df = df.set_index('timestamp')
df['date'] = df.index.map(lambda t: t.date())
df['time'] = df.index.map(lambda t: t.time())
df_pivot = pd.pivot_table(df, values='value', index='timestamp', columns='time')
df_pivot = df_pivot.fillna(0.0)
print(df_pivot)
Generated output
time 00:00:00 00:15:00 00:30:00 00:45:00 01:00:00 01:15:00 01:30:00 01:45:00
timestamp
2008-03-01 00:00:00 55.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2008-03-01 00:15:00 0.0 20.0 0.0 0.0 0.0 0.0 0.0 0.0
2008-03-01 00:30:00 0.0 0.0 13.0 0.0 0.0 0.0 0.0 0.0
2008-03-01 00:45:00 0.0 0.0 0.0 78.0 0.0 0.0 0.0 0.0
2008-03-01 01:00:00 0.0 0.0 0.0 0.0 34.0 0.0 0.0 0.0
2008-03-01 01:15:00 0.0 0.0 0.0 0.0 0.0 123.0 0.0 0.0
2008-03-01 01:30:00 0.0 0.0 0.0 0.0 0.0 0.0 25.0 0.0
2008-03-01 01:45:00 0.0 0.0 0.0 0.0 0.0 0.0 0.0 91.0
2008-03-02 00:00:00 55.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2008-03-02 00:15:00 0.0 46.0 0.0 0.0 0.0 0.0 0.0 0.0
2008-03-02 00:30:00 0.0 0.0 66.0 0.0 0.0 0.0 0.0 0.0
2008-03-02 00:45:00 0.0 0.0 0.0 24.0 0.0 0.0 0.0 0.0
2008-03-02 01:00:00 0.0 0.0 0.0 0.0 70.0 0.0 0.0 0.0
2008-03-02 01:15:00 0.0 0.0 0.0 0.0 0.0 32.0 0.0 0.0
2008-03-02 01:30:00 0.0 0.0 0.0 0.0 0.0 0.0 15.0 0.0
2008-03-02 01:45:00 0.0 0.0 0.0 0.0 0.0 0.0 0.0 92.0
I want to select e.g., only the data for 2008-03-01 00:00:00, 2008-03-01 01:15:00, and 2008-03-02 01:00:00.
Expected output
time 00:00:00 00:15:00 00:30:00 00:45:00 01:00:00 01:15:00 01:30:00 01:45:00
timestamp
2008-03-01 00:00:00 55.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2008-03-01 01:15:00 0.0 0.0 0.0 0.0 0.0 123.0 0.0 0.0
2008-03-02 01:00:00 0.0 0.0 0.0 0.0 70.0 0.0 0.0 0.0
How can I do that
Use list of datetimes converted by to_datetime and select by DataFrame.loc:
#create DatetimeIndex
df = pd.read_csv('df.csv', index_col='timestamp', parse_dates=['timestamp'])
#used pandas methods
df['date'] = df.index.date
df['time'] = df.index.time
#added fill_value parameter
df_pivot = pd.pivot_table(df,values='value',index='timestamp',columns='time',fill_value=0)
L = ['2008-03-01 00:00:00','2008-03-01 01:15:00','2008-03-02 01:00:00']
df = df_pivot.loc[pd.to_datetime(L)]
print (df)
time 00:00:00 00:15:00 00:30:00 00:45:00 01:00:00 \
2008-03-01 00:00:00 55 0 0 0 0
2008-03-01 01:15:00 0 0 0 0 0
2008-03-02 01:00:00 0 0 0 0 70
time 01:15:00 01:30:00 01:45:00
2008-03-01 00:00:00 0 0 0
2008-03-01 01:15:00 123 0 0
2008-03-02 01:00:00 0 0 0
Input:
#Fixed-mono-cell temperature
parameters = pvlib.temperature.TEMPERATURE_MODEL_PARAMETERS['sapm']['open_rack_glass_glass'] #to extract specfic parameter
cell_temperature_mono_fixed = pvlib.temperature.sapm_cell(effective_irrad_mono_fixed,
df['T_a'],
df['W_s'],
**parameters)
cell_temperature_mono_fixed
Output:
2005-01-01 01:00:00 NaN
2005-01-01 02:00:00 NaN
2005-01-01 03:00:00 NaN
2005-01-01 04:00:00 NaN
2005-01-01 05:00:00 NaN
..
8755 NaN
8756 NaN
8757 NaN
8758 NaN
8759 NaN
Length: 17520, dtype: float64
cell_temperature_mono_fixed.plot
Output:
/Users/charlielinck/opt/anaconda3/lib/python3.9/site-packages/pandas/core/indexes/base.py:4024: RuntimeWarning: '
Extra data information:
df: dataframe
date_time Sun_Az Sun_alt GHI DHI DNI T_a W_s
0 2005-01-01 01:00:00 17.9 90.0 0.0 0.0 0.0 15.5 13.3
1 2005-01-01 02:00:00 54.8 90.0 0.0 0.0 0.0 17.0 14.5
2 2005-01-01 03:00:00 73.7 90.0 0.0 0.0 0.0 16.7 14.0
3 2005-01-01 04:00:00 85.7 90.0 0.0 0.0 0.0 16.7 14.2
4 2005-01-01 05:00:00 94.9 90.0 0.0 0.0 0.0 16.7 14.1
5 2005-01-01 06:00:00 103.5 90.0 0.0 0.0 0.0 16.6 14.3
6 2005-01-01 07:00:00 111.6 90.0 0.0 0.0 0.0 16.5 13.8
7 2005-01-01 08:00:00 120.5 89.6 1.0 1.0 0.0 16.6 16.0
8 2005-01-01 09:00:00 130.5 79.9 27.0 27.0 0.0 16.8 16.5
9 2005-01-01 10:00:00 141.8 71.7 55.0 55.0 0.0 16.9 16.9
10 2005-01-01 11:00:00 154.9 65.5 83.0 83.0 0.0 17.0 17.2
11 2005-01-01 12:00:00 169.8 61.9 114.0 114.0 0.0 17.4 17.9
12 2005-01-01 13:00:00 185.2 61.4 110.0 110.0 0.0 17.5 18.0
13 2005-01-01 14:00:00 200.4 64.0 94.0 94.0 0.0 17.5 17.8
14 2005-01-01 15:00:00 214.3 69.5 70.0 70.0 0.0 17.5 17.6
15 2005-01-01 16:00:00 226.3 77.2 38.0 38.0 0.0 17.2 17.0
16 2005-01-01 17:00:00 236.5 86.4 4.0 4.0 0.0 16.7 16.3
17 2005-01-01 18:00:00 245.5 90.0 0.0 0.0 0.0 16.0 14.5
18 2005-01-01 19:00:00 254.2 90.0 0.0 0.0 0.0 14.9 13.0
19 2005-01-01 20:00:00 262.3 90.0 0.0 0.0 0.0 16.0 14.1
20 2005-01-01 21:00:00 271.3 90.0 0.0 0.0 0.0 15.1 13.3
21 2005-01-01 22:00:00 282.1 90.0 0.0 0.0 0.0 15.5 13.2
22 2005-01-01 23:00:00 298.1 90.0 0.0 0.0 0.0 15.6 13.0
23 2005-01-02 00:00:00 327.5 90.0 0.0 0.0 0.0 15.8 13.1
df['T_a'] is temperature data,
df['W_s'] is windspeed data
effective_irrad_mono_fixed.head(24)
date_time
2005-01-01 01:00:00 0.000000
2005-01-01 02:00:00 0.000000
2005-01-01 03:00:00 0.000000
2005-01-01 04:00:00 0.000000
2005-01-01 05:00:00 0.000000
2005-01-01 06:00:00 0.000000
2005-01-01 07:00:00 0.000000
2005-01-01 08:00:00 0.936690
2005-01-01 09:00:00 25.168996
2005-01-01 10:00:00 51.165091
2005-01-01 11:00:00 77.354266
2005-01-01 12:00:00 108.002486
2005-01-01 13:00:00 103.809820
2005-01-01 14:00:00 88.138705
2005-01-01 15:00:00 65.051870
2005-01-01 16:00:00 35.390518
2005-01-01 17:00:00 3.742581
2005-01-01 18:00:00 0.000000
2005-01-01 19:00:00 0.000000
2005-01-01 20:00:00 0.000000
2005-01-01 21:00:00 0.000000
2005-01-01 22:00:00 0.000000
2005-01-01 23:00:00 0.000000
2005-01-02 00:00:00 0.000000
Question: I don't understand that if I simply run the function I only get NaN values, might it have something to with the timestamp.
I believe this also results in the RunTimeWarning when I want to plot the function.
This is not really a pvlib issue, more a pandas issue. The problem is that your input time series objects are not on a consistent index: the irradiance input has a pandas.DatetimeIndex while the temperature and wind speed inputs have pandas.RangeIndex (see the index printed out from your df). Math operations on Series are done by aligning index elements and substituting NaN where things don't line up. For example see how only the shared index elements correspond to non-NaN values here:
In [46]: a = pd.Series([1, 2, 3], index=[1, 2, 3])
...: b = pd.Series([2, 3, 4], index=[2, 3, 4])
...: a*b
Out[46]:
1 NaN
2 4.0
3 9.0
4 NaN
dtype: float64
If you examine the index of your cell_temperature_mono_fixed, you'll see it has both timestamps (from the irradiance input) and integers (from the other two), so it's taking the union of the indexes but only filling in values for the intersection (which is empty in this case).
So to fix your problem, you should make sure all the inputs are on a consistent index. The easiest way to do that is probably at the dataframe level, i.e. df = df.set_index('date_time').
I have a timeseries with data related to the irradiance of the sun. I have data for every hour during a year, but every month has data from a diferent year. For example, the data taken in March can be from 2012 and the data taken in January can be from 2014.
T2m RH G(h) Gb(n) Gd(h) IR(h) WS10m WD10m SP Hour Month
time(UTC)
2012-01-01 00:00:00 16.00 81.66 0.0 -0.0 0.0 310.15 2.56 284.0 102252.0 0 1
2012-01-01 01:00:00 15.97 82.42 0.0 -0.0 0.0 310.61 2.49 281.0 102228.0 1 1
2012-01-01 02:00:00 15.93 83.18 0.0 -0.0 0.0 311.06 2.41 278.0 102205.0 2 1
2012-01-01 03:00:00 15.89 83.94 0.0 -0.0 0.0 311.52 2.34 281.0 102218.0 3 1
2012-01-01 04:00:00 15.85 84.70 0.0 -0.0 0.0 311.97 2.26 284.0 102232.0 4 1
... ... ... ... ... ... ... ... ... ... ... ...
2011-12-31 19:00:00 16.19 77.86 0.0 -0.0 0.0 307.88 2.94 301.0 102278.0 19 12
2011-12-31 20:00:00 16.15 78.62 0.0 -0.0 0.0 308.33 2.86 302.0 102295.0 20 12
2011-12-31 21:00:00 16.11 79.38 0.0 -0.0 0.0 308.79 2.79 297.0 102288.0 21 12
2011-12-31 22:00:00 16.08 80.14 0.0 -0.0 0.0 309.24 2.71 292.0 102282.0 22 12
2011-12-31 23:00:00 16.04 80.90 0.0 -0.0 0.0 309.70 2.64 287.0 102275.0 23 12
My question is: there is a way I can set all the data to a certain year?
For example, set all data to 2014
T2m RH G(h) Gb(n) Gd(h) IR(h) WS10m WD10m SP Hour Month
time(UTC)
2014-01-01 00:00:00 16.00 81.66 0.0 -0.0 0.0 310.15 2.56 284.0 102252.0 0 1
2014-01-01 01:00:00 15.97 82.42 0.0 -0.0 0.0 310.61 2.49 281.0 102228.0 1 1
2014-01-01 02:00:00 15.93 83.18 0.0 -0.0 0.0 311.06 2.41 278.0 102205.0 2 1
2014-01-01 03:00:00 15.89 83.94 0.0 -0.0 0.0 311.52 2.34 281.0 102218.0 3 1
2014-01-01 04:00:00 15.85 84.70 0.0 -0.0 0.0 311.97 2.26 284.0 102232.0 4 1
... ... ... ... ... ... ... ... ... ... ... ...
2014-12-31 19:00:00 16.19 77.86 0.0 -0.0 0.0 307.88 2.94 301.0 102278.0 19 12
2014-12-31 20:00:00 16.15 78.62 0.0 -0.0 0.0 308.33 2.86 302.0 102295.0 20 12
2014-12-31 21:00:00 16.11 79.38 0.0 -0.0 0.0 308.79 2.79 297.0 102288.0 21 12
2014-12-31 22:00:00 16.08 80.14 0.0 -0.0 0.0 309.24 2.71 292.0 102282.0 22 12
2014-12-31 23:00:00 16.04 80.90 0.0 -0.0 0.0 309.70 2.64 287.0 102275.0 23 12
Thanks in advance.
Use offsets.DateOffset with year (without s) for set same year in all DatetimeIndex:
rng = pd.date_range('2009-04-03', periods=10, freq='350D')
df = pd.DataFrame({ 'a': range(10)}, rng)
print (df)
a
2009-04-03 0
2010-03-19 1
2011-03-04 2
2012-02-17 3
2013-02-01 4
2014-01-17 5
2015-01-02 6
2015-12-18 7
2016-12-02 8
2017-11-17 9
df.index += pd.offsets.DateOffset(year=2014)
print (df)
a
2014-04-03 0
2014-03-19 1
2014-03-04 2
2014-02-17 3
2014-02-01 4
2014-01-17 5
2014-01-02 6
2014-12-18 7
2014-12-02 8
2014-11-17 9
Another idea with Index.map and replace:
df.index = df.index.map(lambda x: x.replace(year=2014))
I have a timeseries
date
2009-12-23 0.0
2009-12-28 0.0
2009-12-29 0.0
2009-12-30 0.0
2009-12-31 0.0
2010-01-04 0.0
2010-01-05 0.0
2010-01-06 0.0
2010-01-07 0.0
2010-01-08 0.0
2010-01-11 0.0
2010-01-12 0.0
2010-01-13 0.0
2010-01-14 0.0
2010-01-15 0.0
2010-01-18 0.0
2010-01-19 0.0
2010-01-20 0.0
2010-01-21 0.0
2010-01-22 0.0
2010-01-25 0.0
2010-01-26 0.0
2010-01-27 0.0
2010-01-28 0.0
2010-01-29 0.0
2010-02-01 0.0
2010-02-02 0.0
I would like to set the value to 1 based on the following rule:
If the constant is set 9 this means the 9th of each month. Due to
that that 2010-01-09 doesn't exist I would like to set the next date
that exists in the series to 1 which is 2010-01-11 above.
I have tried to create two series one (series1) with day < 9 set to 1 and one (series2) with day > 9 to 1 and then series1.shift(1) * series2
It works in the middle of the month but not if day is set to 1 due to that the last date in previous month is set to 0 in series1.
Assume your timeseries is s with a datetimeindex
I want to create a groupby object of all index values whose days are greater than or equal to 9.
g = s.index.to_series().dt.day.ge(9).groupby(pd.TimeGrouper('M'))
Then I'll check that there is at least one day past >= 9 and grab the first among them. With those, I'll assign the value of 1.
s.loc[g.idxmax()[g.any()]] = 1
s
date
2009-12-23 1.0
2009-12-28 0.0
2009-12-29 0.0
2009-12-30 0.0
2009-12-31 0.0
2010-01-04 0.0
2010-01-05 0.0
2010-01-06 0.0
2010-01-07 0.0
2010-01-08 0.0
2010-01-11 1.0
2010-01-12 0.0
2010-01-13 0.0
2010-01-14 0.0
2010-01-15 0.0
2010-01-18 0.0
2010-01-19 0.0
2010-01-20 0.0
2010-01-21 0.0
2010-01-22 0.0
2010-01-25 0.0
2010-01-26 0.0
2010-01-27 0.0
2010-01-28 0.0
2010-01-29 0.0
2010-02-01 0.0
2010-02-02 0.0
Name: val, dtype: float64
Note that 2009-12-23 also was assigned a 1 as it satisfies this requirement as well.
I have a dataframe df with this structure :
TIMESTAMP probab-activ1 probab-activ3 probab-activ5
2015-07-31 23:00:00 90.0 90.0 90.0
2015-07-31 23:10:00 0.0 0.0 0.0
2015-07-31 23:20:00 0.0 0.0 0.0
2015-07-31 23:30:00 0.0 0.0 0.0
2015-07-31 23:40:00 0.0 0.0 0.0
...
2015-10-31 23:20:00 0.0 0.0 0.0
2015-10-31 23:30:00 0.0 0.0 0.0
2015-10-31 23:40:00 0.0 0.0 0.0
I need to calculate for each day of the week (monday , tuesday ,.., sunday) the mean of the probability (probab-activ1, probab-activ3 and probab-activ5) durant the 2 last months.
Any idea to solve this problem?
Thank you in advance
You can use the datetime module and convert your timestamp to a format that is useful for your purpose. For example, you could do:
import datetime
timestamp = '2015-07-31 23:00:00'
day_of_week = datetime.datetime.strptime(timestamp, '%Y-%m-%d %H:%M:%S').strftime('%a')
day_of_week
'Fri'