Develop Reference

Transpose hourly data in rows to new columns Pandas

I have a table recorded hourly weather data like this:
Year Month Day Time Temp
Date/Time
2005-01-01 00:00:00 2005 1 1 0:00 6.0
2005-01-01 01:00:00 2005 1 1 1:00 6.1
2005-01-01 02:00:00 2005 1 1 2:00 6.7
2005-01-01 03:00:00 2005 1 1 3:00 6.8
2005-01-01 04:00:00 2005 1 1 4:00 6.3
2005-01-01 05:00:00 2005 1 1 5:00 6.6
2005-01-01 06:00:00 2005 1 1 6:00 6.9
2005-01-01 07:00:00 2005 1 1 7:00 7.1
2005-01-01 08:00:00 2005 1 1 8:00 6.9
2005-01-01 09:00:00 2005 1 1 9:00 6.7
2005-01-01 10:00:00 2005 1 1 10:00 7.1
2005-01-01 11:00:00 2005 1 1 11:00 7.1
2005-01-01 12:00:00 2005 1 1 12:00 7.2
2005-01-01 13:00:00 2005 1 1 13:00 7.7
2005-01-01 14:00:00 2005 1 1 14:00 8.8
2005-01-01 15:00:00 2005 1 1 15:00 8.6
2005-01-01 16:00:00 2005 1 1 16:00 7.4
2005-01-01 17:00:00 2005 1 1 17:00 6.8
2005-01-01 18:00:00 2005 1 1 18:00 6.3
2005-01-01 19:00:00 2005 1 1 19:00 5.9
2005-01-01 20:00:00 2005 1 1 20:00 5.6
2005-01-01 21:00:00 2005 1 1 21:00 3.6
2005-01-01 22:00:00 2005 1 1 22:00 2.6
2005-01-01 23:00:00 2005 1 1 23:00 1.7
I wanted to save the dataframe in this format:
How can I transpose the dataframe and create new columns for each record?

1) Copy below data
Date/Time Year Month Day Time Temp
0 1/1/2005 0:00 2005 1 1 0:00 6.0
1 1/1/2005 1:00 2005 1 1 1:00 6.1
2 1/1/2005 2:00 2005 1 1 2:00 6.7
3 1/1/2005 3:00 2005 1 1 3:00 6.8
4 1/1/2005 4:00 2005 1 1 4:00 6.3
5 1/1/2005 5:00 2005 1 1 5:00 6.6
6 1/1/2005 6:00 2005 1 1 6:00 6.9
7 1/1/2005 7:00 2005 1 1 7:00 7.1
8 1/1/2005 8:00 2005 1 1 8:00 6.9
9 1/1/2005 9:00 2005 1 1 9:00 6.7
10 1/1/2005 10:00 2005 1 1 10:00 7.1
11 1/1/2005 11:00 2005 1 1 11:00 7.1
12 1/1/2005 12:00 2005 1 1 12:00 7.2
2) Use pd.read_clipboard with a double space or more param due to space in Date/Time column
import pandas as pd
df=pd.read_clipboard('\s\s+')
df
3) format date/time columns and create a pivot table and reset/rename axis.
df['Date/Time']=pd.to_datetime(df['Date/Time'],format='%m/%d/%Y
%H:%M').dt.strftime('%m/%d/%Y')
df['Time']=pd.to_datetime(df['Time']).dt.time
df=pd.pivot_table(df, index='Date/Time', columns='Time', values='Temp').reset_index().rename_axis(index=None, columns=None)
df['Date/Time']=df['Date/Time'].apply(lambda x:(x + ' 0:00'))
df
Output:
Date/Time 00:00:00 01:00:00 02:00:00 03:00:00 04:00:00 05:00:00 06:00:00 07:00:00 08:00:00 09:00:00 10:00:00 11:00:00 12:00:00
01/01/2005 6.0 6.1 6.7 6.8 6.3 6.6 6.9 7.1 6.9 6.7 7.1 7.1 7.2

This does the trick similarly to previous answer but on a new date column such that only one row per day is created:
datetimes = pd.date_range("2005-01-01 00:00:00","2005-01-02 23:00:00", freq="1h")
df = pd.DataFrame({"Date/Time": datetimes, "temp": rand(len(datetimes))})
df["Date"] = df["Date/Time"].dt.date
df["Hour"] = df["Date/Time"].dt.hour
reshaped= df.pivot(index='Date', columns='Hour', values='temp')
reshaped.columns = ['HR'+str(hour) for hour in reshaped.columns]

How to combine 3 separate columns of year(2 digit) ,month and day into single date column

I am combining 3 seoerate columns of year,month and day into a single column of my dataframe. But the year is in 2 digit which is giving error.
I have tried to_datetime() to do the same in jupyter notebook
Dataframe is in this form:
Yr Mo Dy RPT VAL ROS KIL SHA BIR DUB CLA MUL CLO BEL
61 1 1 15.04 14.96 13.17 9.29 NaN 9.87 13.67 10.25 10.83 12.58 18.50
61 1 2 14.71 NaN 10.83 6.50 12.62 7.67 11.50 10.04 9.79 9.67 17.54
61 1 3 18.50 16.88 12.33 10.13 11.17 6.17 11.25 NaN 8.50 7.67 12.75
data.rename(columns={'Yr':'Year','Mo':'Month','Dy':'Day'},inplace=True)
data['Date']=pd.to_datetime(data[['Year','Month','Day']],format='%y%m%d')
The error i am getting is:
cannot assemble the datetimes: time data 610101 does not match format '%Y%m%d' (match)

There is problem to_datetime with specify columns ['Year','Month','Day'] need YYYY format, so need alternative solution, because year is YY only:
s = data[['Yr','Mo','Dy']].astype(str).apply('-'.join, 1)
data['Date'] = pd.to_datetime(s, format='%y-%m-%d')
print (data)
Yr Mo Dy RPT VAL ROS KIL SHA BIR DUB CLA MUL \
0 61 1 1 15.04 14.96 13.17 9.29 NaN 9.87 13.67 10.25 10.83
1 61 1 2 14.71 NaN 10.83 6.50 12.62 7.67 11.50 10.04 9.79
2 61 1 3 18.50 16.88 12.33 10.13 11.17 6.17 11.25 NaN 8.50
CLO BEL Date
0 12.58 18.50 2061-01-01
1 9.67 17.54 2061-01-02
2 7.67 12.75 2061-01-03

Pandas Dataframe Pivot and reindex n timeseries

I have a pandas dataframe containing n time series in the same Datetime column, each one associated to a different Id, with a corresponding value associated. I would like to pivot the table and reindex to the nearest timestamp. Notice that there can be cases where a timestamp is missing, as in Id-3, in this case the value would need to become NaN.
Datetime Id Value
5-26-17 8:00 1 2.3
5-26-17 8:30 1 4.5
5-26-17 9:00 1 7
5-26-17 9:30 1 8.1
5-26-17 10:00 1 7.9
5-26-17 10:30 1 3.4
5-26-17 11:00 1 2.1
5-26-17 11:30 1 1.8
5-26-17 12:00 1 0.4
5-26-17 8:02 2 2.6
5-26-17 8:32 2 4.8
5-26-17 9:02 2 7.3
5-26-17 9:32 2 8.4
5-26-17 10:02 2 8.2
5-26-17 10:32 2 3.7
5-26-17 11:02 2 2.4
5-26-17 11:32 2 2.1
5-26-17 12:02 2 0.7
5-26-17 8:30 3 4.5
5-26-17 9:00 3 7
5-26-17 9:30 3 8.1
5-26-17 10:00 3 7.9
5-26-17 10:30 3 3.4
5-26-17 11:00 3 2.1
5-26-17 11:30 3 1.8
5-26-17 12:00 3 0.4
Expected results:
Datetime Id-1 Id-2 Id-3
5-26-17 8:00 2.3 2.6 NaN
5-26-17 8:30 4.5 4.8 4.5
5-26-17 9:00 7 7.3 7
5-26-17 9:30 8.1 8.4 8.1
5-26-17 10:00 7.9 8.2 7.9
5-26-17 10:30 3.4 3.7 3.4
5-26-17 11:00 2.1 2.4 2.1
5-26-17 11:30 1.8 2.1 1.8
5-26-17 12:00 0.4 0.7 0.4
How would you do this?

I believe need convert column to datetimes and floor by 30 minutes by floor, last pivot and add_prefix:
df['Datetime'] = pd.to_datetime(df['Datetime']).dt.floor('30T')
df = df.pivot('Datetime','Id','Value').add_prefix('Id-')
print (df)
Id Id-1 Id-2 Id-3
Datetime
2017-05-26 08:00:00 2.3 2.6 NaN
2017-05-26 08:30:00 4.5 4.8 4.5
2017-05-26 09:00:00 7.0 7.3 7.0
2017-05-26 09:30:00 8.1 8.4 8.1
2017-05-26 10:00:00 7.9 8.2 7.9
2017-05-26 10:30:00 3.4 3.7 3.4
2017-05-26 11:00:00 2.1 2.4 2.1
2017-05-26 11:30:00 1.8 2.1 1.8
2017-05-26 12:00:00 0.4 0.7 0.4
Another solution is use resample with mean:
df['Datetime'] = pd.to_datetime(df['Datetime'])
df = (df.set_index('Datetime')
.groupby('Id')
.resample('30T')['Value']
.mean().unstack(0)
.add_prefix('Id-'))
print (df)
Id Id-1 Id-2 Id-3
Datetime
2017-05-26 08:00:00 2.3 2.6 NaN
2017-05-26 08:30:00 4.5 4.8 4.5
2017-05-26 09:00:00 7.0 7.3 7.0
2017-05-26 09:30:00 8.1 8.4 8.1
2017-05-26 10:00:00 7.9 8.2 7.9
2017-05-26 10:30:00 3.4 3.7 3.4
2017-05-26 11:00:00 2.1 2.4 2.1
2017-05-26 11:30:00 1.8 2.1 1.8
2017-05-26 12:00:00 0.4 0.7 0.4

Average pandas dataframe on time index for a particular time interval

I have a dataframe where for each timestamp there are some points earned by the user. It looks like the following i.e. data was collected after few seconds
>> df.head()
points
timestamp
2017-05-29 17:40:45 5
2017-05-29 17:41:53 7
2017-05-29 17:42:34 3
2017-05-29 17:42:36 8
2017-05-29 17:42:37 6
Then I wanted to resample it to an interval of 5 minutes so I did this
>> df.resample("5min").mean()
points
timestamp
5/29/2017 17:40 8
5/29/2017 17:45 1
5/29/2017 17:50 4
5/29/2017 17:55 3
5/29/2017 18:00 8
5/30/2017 17:30 3
5/30/2017 17:35 3
5/30/2017 17:40 7
5/30/2017 17:45 8
5/30/2017 17:50 5
5/30/2017 17:55 7
5/30/2017 18:00 1
Now I want to give an input like this input_time = "17:00-18:00" and I want to divide the input time into 5min interval for e.g. [17:05, 17:10 ... 17:55, 18:00]. After that for each interval I want to get the average points earned for that particular time interval. The results should look like the following
interval points
17:00 -
17:05 -
….
17:30 3
17:35 3
17:40 7.5
17:45 4.5
17:50 4.5
17:55 5
18:00 4.5
Need your help. Thanks

Create DatetimeIndex by date_range and change format by strftime:
input_time = "17:00-18:00"
s,e = input_time.split('-')
r = pd.date_range(s, e, freq='5T').strftime('%H:%M')
print (r)
['17:00' '17:05' '17:10' '17:15' '17:20' '17:25' '17:30' '17:35' '17:40'
'17:45' '17:50' '17:55' '18:00']
Also convert original index for groupby with aggregate mean, last reindex by range:
df = df.groupby(df.index.strftime('%H:%M'))['points'].mean().reindex(r)
print (df)
17:00 NaN
17:05 NaN
17:10 NaN
17:15 NaN
17:20 NaN
17:25 NaN
17:30 3.0
17:35 3.0
17:40 7.5
17:45 4.5
17:50 4.5
17:55 5.0
18:00 4.5
Name: points, dtype: float64

Pandas or numpy loop to find the 12 hour rolling minimum of each column and output it as a new column

How can I use pandas or numpy to do this? I need to calculate a rolling window (minimum) of station data, and either add it as a new column or output it to a new dataframe all together.
I currently have the following code:
dataframe data:
stamp year month day time 10805 2855 3172 10888 7485 3088 2896 7550 46367 2925 51739 3231 2942 10795 10724 10667 10727 2967 27476 3244 44344 3318 28011 3193 10784 3328 50091 3259 3157 10821 10785 10660 49808 44203 3062 31408 32149 10915 31407 2180 26971 32254 2224 32230 27212 32256 10708 10225 32257 32395 32313 2263 2265 1920 31411 2273 8804 32433 10725 2312 10907 10889 10882 3721 50821 3471 27741 26857 26866 29886 8993 7333 3605 3853 10185 29593 10881 3649 10884 27119 10927 10188 3562 3698
1/1/1994 0:00 1994 1 1 0:00 -20.5 -15.3 -16.4 -14.9 -21.3 -22.8 -24.4 -5.5 -19.7 -8.4 -38.6 -28.7 -30.3
1/1/1994 1:00 1994 1 1 1:00 -20.3 -15.2 -16.2 -14.8 -21.5 -22.4 -23.6 -5.3 -19.8 -9.3 -38.4 -26.7 -31.1
1/1/1994 2:00 1994 1 1 2:00 -20.6 -15.2 -17.4 -14.1 -22 -22.8 -23.2 -4.8 -20 -8.8 -38.7 -26 -31.3
1/1/1994 3:00 1994 1 1 3:00 -19.7 -15 -18.3 -14.3 -22.1 -22.6 -22.9 -4.9 -20.3 -9.4 -38.9 -25.6 -32.4
1/1/1994 4:00 1994 1 1 4:00 -19.2 -15.4 -18.2 -14.5 -22.5 -22.5 -23.3 -4.8 -20.3 -8.4 -38.4 -26 -31.1
1/1/1994 5:00 1994 1 1 5:00 -18.8 -16.3 -18.4 -14.4 -22.5 -22.5 -23.3 -5.1 -20.9 -8.1 -30.5 -26 -30.7
1/1/1994 6:00 1994 1 1 6:00 -18.6 -17 -18.9 -14.3 -23.2 -22.8 -23.2 -3 -25.8 -9.3 -30.8 -25.6 -30.3
1/1/1994 7:00 1994 1 1 7:00 -18.6 -16.6 -19.4 -14 -23.6 -22.9 -23.3 -7.2 -25 -10.2 -30.4 -25.1 -30.1
1/1/1994 8:00 1994 1 1 8:00 -18.7 -16 -19.5 -14 -24.1 -23.1 -23.3 -7.8 -24.6 -10.9 -30.3 -24.5 -29.3
1/1/1994 9:00 1994 1 1 9:00 -18.9 -16.4 -19.9 -14.3 -24.1 -23.1 -24 -8.2 -24.5 -10.7 -31.4 -24.4 -29
1/1/1994 10:00 1994 1 1 10:00 -18.9 -16.5 -19.6 -14.5 -24.2 -23.7 -24.7 -8.2 -24.1 -10.5 -31 -24.1 -27.9
1/1/1994 11:00 1994 1 1 11:00 -18.7 -16.3 -19.6 -15.2 -24.3 -23.7 -24.9 -8.3 -23.9 -10.8 -31 -23.8 -27
The data exists in a .csv file (this is just another view of the same dataset above).
stamp,year,month,day,time,10805,2855,3172,10888,7485,3088,2896,7550,46367,2925,51739,3231,2942,10795,10724,10667,10727,2967,27476,3244,44344,3318,28011,3193,10784,3328,50091,3259,3157,10821,10785,10660,49808,44203,3062,31408,32149,10915,31407,2180,26971,32254,2224,32230,27212,32256,10708,10225,32257,32395,32313,2263,2265,1920,31411,2273,8804,32433,10725,2312,10907,10889,10882,3721,50821,3471,27741,26857,26866,29886,8993,7333,3605,3853,10185,29593,10881,3649,10884,27119,10927,10188,3562,3698
"1994-01-01 00:00","1994","01","01","00:00",,"-20.5",,,,,"-15.3",,,,,"-16.4",,,,,,"-14.9",,"-21.3",,,,,,"-22.8",,,,,,,,,"-24.4",,,,,,,,,,,,,,,,,"-5.5",,"-19.7",,"-8.4",,,,,,,,"-38.6",,"-28.7",,,,,,,,,,,,,,,,,,"-30.3"
"1994-01-01 01:00","1994","01","01","01:00",,"-20.3",,,,,"-15.2",,,,,"-16.2",,,,,,"-14.8",,"-21.5",,,,,,"-22.4",,,,,,,,,"-23.6",,,,,,,,,,,,,,,,,"-5.3",,"-19.8",,"-9.3",,,,,,,,"-38.4",,"-26.7",,,,,,,,,,,,,,,,,,"-31.1"
"1994-01-01 02:00","1994","01","01","02:00",,"-20.6",,,,,"-15.2",,,,,"-17.4",,,,,,"-14.1",,"-22.0",,,,,,"-22.8",,,,,,,,,"-23.2",,,,,,,,,,,,,,,,,"-4.8",,"-20.0",,"-8.8",,,,,,,,"-38.7",,"-26.0",,,,,,,,,,,,,,,,,,"-31.3"
"1994-01-01 03:00","1994","01","01","03:00",,"-19.7",,,,,"-15.0",,,,,"-18.3",,,,,,"-14.3",,"-22.1",,,,,,"-22.6",,,,,,,,,"-22.9",,,,,,,,,,,,,,,,,"-4.9",,"-20.3",,"-9.4",,,,,,,,"-38.9",,"-25.6",,,,,,,,,,,,,,,,,,"-32.4"
"1994-01-01 04:00","1994","01","01","04:00",,"-19.2",,,,,"-15.4",,,,,"-18.2",,,,,,"-14.5",,"-22.5",,,,,,"-22.5",,,,,,,,,"-23.3",,,,,,,,,,,,,,,,,"-4.8",,"-20.3",,"-8.4",,,,,,,,"-38.4",,"-26.0",,,,,,,,,,,,,,,,,,"-31.1"
"1994-01-01 05:00","1994","01","01","05:00",,"-18.8",,,,,"-16.3",,,,,"-18.4",,,,,,"-14.4",,"-22.5",,,,,,"-22.5",,,,,,,,,"-23.3",,,,,,,,,,,,,,,,,"-5.1",,"-20.9",,"-8.1",,,,,,,,"-30.5",,"-26.0",,,,,,,,,,,,,,,,,,"-30.7"
"1994-01-01 06:00","1994","01","01","06:00",,"-18.6",,,,,"-17.0",,,,,"-18.9",,,,,,"-14.3",,"-23.2",,,,,,"-22.8",,,,,,,,,"-23.2",,,,,,,,,,,,,,,,,"-3.0",,"-25.8",,"-9.3",,,,,,,,"-30.8",,"-25.6",,,,,,,,,,,,,,,,,,"-30.3"
"1994-01-01 07:00","1994","01","01","07:00",,"-18.6",,,,,"-16.6",,,,,"-19.4",,,,,,"-14.0",,"-23.6",,,,,,"-22.9",,,,,,,,,"-23.3",,,,,,,,,,,,,,,,,"-7.2",,"-25.0",,"-10.2",,,,,,,,"-30.4",,"-25.1",,,,,,,,,,,,,,,,,,"-30.1"
"1994-01-01 08:00","1994","01","01","08:00",,"-18.7",,,,,"-16.0",,,,,"-19.5",,,,,,"-14.0",,"-24.1",,,,,,"-23.1",,,,,,,,,"-23.3",,,,,,,,,,,,,,,,,"-7.8",,"-24.6",,"-10.9",,,,,,,,"-30.3",,"-24.5",,,,,,,,,,,,,,,,,,"-29.3"
"1994-01-01 09:00","1994","01","01","09:00",,"-18.9",,,,,"-16.4",,,,,"-19.9",,,,,,"-14.3",,"-24.1",,,,,,"-23.1",,,,,,,,,"-24.0",,,,,,,,,,,,,,,,,"-8.2",,"-24.5",,"-10.7",,,,,,,,"-31.4",,"-24.4",,,,,,,,,,,,,,,,,,"-29.0"
"1994-01-01 10:00","1994","01","01","10:00",,"-18.9",,,,,"-16.5",,,,,"-19.6",,,,,,"-14.5",,"-24.2",,,,,,"-23.7",,,,,,,,,"-24.7",,,,,,,,,,,,,,,,,"-8.2",,"-24.1",,"-10.5",,,,,,,,"-31.0",,"-24.1",,,,,,,,,,,,,,,,,,"-27.9"
"1994-01-01 11:00","1994","01","01","11:00",,"-18.7",,,,,"-16.3",,,,,"-19.6",,,,,,"-15.2",,"-24.3",,,,,,"-23.7",,,,,,,,,"-24.9",,,,,,,,,,,,,,,,,"-8.3",,"-23.9",,"-10.8",,,,,,,,"-31.0",,"-23.8",,,,,,,,,,,,,,,,,,"-27.0"
"1994-01-01 12:00","1994","01","01","12:00",,"-18.6",,,,,"-16.1",,,,,"-19.8",,,,,,"-15.1",,"-24.7",,,,,,"-23.6",,,,,,,,,"-24.8",,,,,,,,,,,,,,,,,"-8.5",,"-23.6",,"-10.9",,,,,,,,"-31.4",,"-23.2",,,,,,,,,,,,,,,,,,"-26.6"
"1994-01-01 13:00","1994","01","01","13:00",,"-18.6",,,,,"-15.9",,,,,"-19.7",,,,,,"-15.4",,"-24.3",,,,,,"-23.9",,,,,,,,,"-24.5",,,,,,,,,,,,,,,,,"-9.3",,"-23.3",,"-11.4",,,,,,,,"-31.9",,"-22.8",,,,,,,,,,,,,,,,,,"-25.9"
"1994-01-01 14:00","1994","01","01","14:00",,"-19.4",,,,,"-15.9",,,,,"-19.6",,,,,,"-15.2",,"-24.4",,,,,,"-24.7",,,,,,,,,"-24.7",,,,,,,,,,,,,,,,,"-10.2",,"-23.4",,"-11.9",,,,,,,,"-33.2",,"-22.6",,,,,,,,,,,,,,,,,,"-24.9"
loads the dataframe
df=pd.read_csv('dataframedata.csv')
df.head()
Filtering the dataset to only have the days of interest for within each year.
df=df[(df.day >= 9) & (df.day <= 20)]
create a list of all the stations in the dataframe while dropping all the other information.
st=list(df)
st=st[5:]
st=pd.DataFrame(st)
How would I go about creating a loop that would use the values from st to select columns to execute the following function (which calculates the running 12 hr minimum for each cell while moving through the column)?
st['newcolumnname'] = df.rolling(12).min()[st]
Ultimately I am trying to calculate the 12 hour rolling minimum for each value at each station (column). With particular interest on the days 9 to 20th of each month.

Your first mistake is creating a DataFrame out of those columns. You can just index df.columns.
st = df.columns[5:].tolist()
Next, convert stamp to datetime -
df['stamp'] = pd.to_datetime(df['stamp'])
Finally, call Rolling.min -
df[st].rolling(12).min()
From your data, it seems recordings are made at neat 1 hour intervals. If this isn't the case, slap on a set_index + resample call first and then find the rolling minimum.
df.set_index('stamp').[st]resample('1H').first().rolling(12).min()
Note that in either case, the result is a dataframe, you cannot put this inside a column.