I have a data frame with a date time index, and I would like to multiply some columns with the number of days in that month.
TUFNWGTP TELFS t070101 t070102 t070103 t070104
TUDIARYDATE
2003-01-03 8155462.672158 2 0 0 0 0
2003-01-04 1735322.527819 1 0 0 0 0
2003-01-04 3830527.482672 2 60 0 0 0
2003-01-02 6622022.995205 4 0 0 0 0
2003-01-09 3068387.344956 1 0 0 0 0
Here, I would like to multiply all the columns starting with t with 31. That is, expected output is
TUFNWGTP TELFS t070101 t070102 t070103 t070104
TUDIARYDATE
2003-01-03 8155462.672158 2 0 0 0 0
2003-01-04 1735322.527819 1 0 0 0 0
2003-01-04 3830527.482672 2 1680 0 0 0
2003-01-02 6622022.995205 4 0 0 0 0
2003-01-09 3068387.344956 1 0 0 0 0
I know that there are some ways using calendar or similar, but given that I'm already using pandas, there must be an easier way - I assume.
There is no such datetime property, but there is an offset M - but I don't know how I would use that without massive inefficiency.
There is now a Series.dt.days_in_month attribute for datetime series. Here is an example based on Jeff's answer.
In [3]: df = pd.DataFrame({'date': pd.date_range('20120101', periods=15, freq='M')})
In [4]: df['year'] = df['date'].dt.year
In [5]: df['month'] = df['date'].dt.month
In [6]: df['days_in_month'] = df['date'].dt.days_in_month
In [7]: df
Out[7]:
date year month days_in_month
0 2012-01-31 2012 1 31
1 2012-02-29 2012 2 29
2 2012-03-31 2012 3 31
3 2012-04-30 2012 4 30
4 2012-05-31 2012 5 31
5 2012-06-30 2012 6 30
6 2012-07-31 2012 7 31
7 2012-08-31 2012 8 31
8 2012-09-30 2012 9 30
9 2012-10-31 2012 10 31
10 2012-11-30 2012 11 30
11 2012-12-31 2012 12 31
12 2013-01-31 2013 1 31
13 2013-02-28 2013 2 28
14 2013-03-31 2013 3 31
pd.tslib.monthrange is an unadvertised / undocumented function that handles the days_in_month calculation (adjusting for leap years). This could/should prob be added as a property to Timestamp/DatetimeIndex.
In [34]: df = DataFrame({'date' : pd.date_range('20120101',periods=15,freq='M') })
In [35]: df['year'] = df['date'].dt.year
In [36]: df['month'] = df['date'].dt.month
In [37]: df['days_in_month'] = df.apply(lambda x: pd.tslib.monthrange(x['year'],x['month'])[1], axis=1)
In [38]: df
Out[38]:
date year month days_in_month
0 2012-01-31 2012 1 31
1 2012-02-29 2012 2 29
2 2012-03-31 2012 3 31
3 2012-04-30 2012 4 30
4 2012-05-31 2012 5 31
5 2012-06-30 2012 6 30
6 2012-07-31 2012 7 31
7 2012-08-31 2012 8 31
8 2012-09-30 2012 9 30
9 2012-10-31 2012 10 31
10 2012-11-30 2012 11 30
11 2012-12-31 2012 12 31
12 2013-01-31 2013 1 31
13 2013-02-28 2013 2 28
14 2013-03-31 2013 3 31
Here is a little clunky hand-made method to get the number of days in a month
import datetime
def days_in_month(dt):
next_month = datetime.datetime(
dt.year + dt.month / 12, dt.month % 12 + 1, 1)
start_month = datetime.datetime(dt.year, dt.month, 1)
td = next_month - start_month
return td.days
For example:
>>> days_in_month(datetime.datetime.strptime('2013-12-12', '%Y-%m-%d'))
31
>>> days_in_month(datetime.datetime.strptime('2013-02-12', '%Y-%m-%d'))
28
>>> days_in_month(datetime.datetime.strptime('2012-02-12', '%Y-%m-%d'))
29
>>> days_in_month(datetime.datetime.strptime('2012-01-12', '%Y-%m-%d'))
31
>>> days_in_month(datetime.datetime.strptime('2013-11-12', '%Y-%m-%d'))
30
I let you figure out how to read your table and do the multiplication yourself :)
import pandas as pd
from pandas.tseries.offsets import MonthEnd
df['dim'] = (pd.to_datetime(df.index) + MonthEnd(0)).dt.day
You can omit pd.to_datetime(), if your index is already DatetimeIndex.
Related
I have the following dataframe:
date
wind (°)
wind (kt)
temp (C°)
humidity(%)
currents (°)
currents (kt)
stemp (C°)
sea_temp_diff
wind_distance_diff
wind_speed_diff
temp_diff
humidity_diff
current_distance_diff
current_speed_diff
8 12018
175.000000
16.333333
25.500000
82.500000
60.000000
0.100000
25.400000
-1.066667
23.333333
-0.500000
-0.333333
-12.000000
160.000000
6.666667e-02
9 12019
180.000000
17.000000
23.344828
79.724138
230.000000
0.100000
23.827586
-0.379310
22.068966
1.068966
0.827586
-7.275862
315.172414
3.449034e+02
10 12020
365.000000
208.653846
24.192308
79.346154
355.769231
192.500000
24.730769
574.653846
1121.923077
1151.153846
1149.346154
-19.538462
1500.000000
1.538454e+03
14 22019
530.357143
372.964286
23.964286
81.964286
1270.714286
1071.560714
735.642857
-533.642857
-327.500000
-356.892857
1.857143
-10.321429
-873.571429
-8.928107e+02
15 22020
216.551724
12.689655
24.517241
81.137931
288.275862
172.565517
196.827586
-171.379310
-8.965517
3.724138
1.413793
-7.137931
-105.517241
-1.722724e+02
16 32019
323.225806
174.709677
25.225806
80.741935
260.000000
161.451613
25.709677
480.709677
486.451613
483.967742
0.387097
153.193548
1044.516129
9.677065e+02
17 32020
351.333333
178.566667
25.533333
78.800000
427.666667
166.666667
26.600000
165.533333
-141.000000
-165.766667
166.633333
158.933333
8.333333
1.500000e-01
18 42017
180.000000
14.000000
27.000000
5000.000000
200.000000
0.400000
25.400000
2.600000
20.000000
-4.000000
0.000000
0.000000
-90.000000
-1.000000e-01
19 42019
694.230769
589.769231
24.038462
69.461538
681.153846
577.046154
26.884615
-1.346154
37.307692
-1.692308
1.500000
4.769231
98.846154
1.538462e-01
20 42020
306.666667
180.066667
24.733333
75.166667
427.666667
166.666667
26.800000
165.066667
205.333333
165.200000
1.100000
-4.066667
360.333333
3.334233e+02
21 52017
146.333333
11.966667
22.900000
5000.000000
116.333333
0.410000
26.066667
-1.553333
8.666667
0.833333
-0.766667
0.000000
95.000000
-1.300000e-01
22 52019
107.741935
12.322581
23.419355
63.032258
129.354839
0.332258
25.935484
-1.774194
14.838710
0.096774
-0.612903
-14.451613
130.967742
I need to sort the 'date' column chronologically, and I'm wondering if there's a way for me to split it two ways, with the '10' in one column and 2017 in another, sort both of them in ascending order, and then bring them back together.
I had tried this:
australia_overview[['month','year']] = australia_overview['date'].str.split("2",expand=True)
But I am getting error like this:
ValueError: Columns must be same length as key
How can I solve this issue?
From your DataFrame :
>>> df = pd.DataFrame({'id': [1, 2, 3, 4],
... 'date': ['1 42018', '12 32019', '8 112020', '23 42021']},
... index = [0, 1, 2, 3])
>>> df
id date
0 1 1 42018
1 2 12 32019
2 3 8 112020
3 4 23 42021
We can split the column to get the first value of day like so :
>>> df['day'] = df['date'].str.split(' ', expand=True)[0]
>>> df
id date day
0 1 1 42018 1
1 2 12 32019 12
2 3 8 112020 8
3 4 23 42021 23
And get the 4 last digit from the column date for the year to get the expected result :
>>> df['year'] = df['date'].str[-4:].astype(int)
>>> df
id date day year
0 1 1 42018 1 2018
1 2 12 32019 12 2019
2 3 8 112020 8 2020
3 4 23 42021 23 2021
Bonus : as asked in the comment, you can even get the month using the same principle :
>>> df['month'] = df['date'].str.split(' ', expand=True)[1].str[:-4].astype(int)
>>> df
id date day year month
0 1 1 42018 1 2018 4
1 2 12 32019 12 2019 3
2 3 8 112020 8 2020 11
3 4 23 42021 23 2021 4
I have this dataset, which have year, month, week and sales numbers:
df = pd.DataFrame()
df['year'] = [2011,2011,2011,2012,2012,2012]
df['month'] = [12,12,12,1,1,1]
df['week'] = [51,52,53,1,2,3]
df['sales'] = [10000,12000,11000,5000,12000,11000]
df['date_ix'] = df['year'] * 1000 + (df['week']-1) * 10 + 1
df['date_week'] = pd.to_datetime(df['date_ix'], format='%Y%W%w')
df
year month week sales date_ix date_week
0 2011 12 51 10000 2011501 2011-12-12
1 2011 12 52 12000 2011511 2011-12-19
2 2011 12 53 11000 2011521 2011-12-26
3 2012 1 1 5000 2012001 2011-12-26
4 2012 1 2 12000 2012011 2012-01-02
5 2012 1 3 11000 2012021 2012-01-09
Now date_week is the beginning day of the week (Monday). I want to convert date_week to day except by the first week of the year, where I want to isolate the day (in this case is 2012-01-01 which was Sunday). I have tried this, but something's wrong.
df['date_start'] = np.where((df['year']==2012) & (df['week']==1), \
pd.to_datetime(str(20120101), format='%Y%m%d'), \
pd.to_datetime(df['date_ix'], format='%Y%W%w'))
year month week sales date_ix date_week date_start
0 2011 12 51 10000 2011501 2011-12-12 1323648000000000000
1 2011 12 52 12000 2011511 2011-12-19 1324252800000000000
2 2011 12 53 11000 2011521 2011-12-26 1324857600000000000
3 2012 1 1 5000 2012001 2011-12-26 2012-01-01 00:00:00
4 2012 1 2 12000 2012011 2012-01-02 1325462400000000000
5 2012 1 3 11000 2012021 2012-01-09 1326067200000000000
The expected result should be:
year month week sales date_ix date_week date_start
0 2011 12 51 10000 2011501 2011-12-12 2011-12-12
1 2011 12 52 12000 2011511 2011-12-19 2011-12-19
2 2011 12 53 11000 2011521 2011-12-26 2011-12-26
3 2012 1 1 5000 2012001 2011-12-26 2012-01-01
4 2012 1 2 12000 2012011 2012-01-02 2012-01-02
5 2012 1 3 11000 2012021 2012-01-09 2012-01-09
Please, any help will be greatly appreciated.
You need to enclose df['year']==2012 and df['week']==1 with parentheses because of the priority of == and &.
df['date_start'] = np.where((df['year']==2012) & (df['week']==1), \
pd.to_datetime(str(20120101), format='%Y%m%d'), \
pd.to_datetime(df['date_ix'], format='%Y%W%w'))
Then change pd.to_datetime(str(20120101), format='%Y%m%d') in np.where to pd.to_datetime(df['year'], format='%Y')
df['date_start'] = np.where((df['year']==2012) & (df['week']==1), \
pd.to_datetime(df['year'], format='%Y'),
df['date_week'])
print(df)
year month week sales date_ix date_week date_start
0 2011 12 51 10000 2011501 2011-12-12 2011-12-12
1 2011 12 52 12000 2011511 2011-12-19 2011-12-19
2 2011 12 53 11000 2011521 2011-12-26 2011-12-26
3 2012 1 1 5000 2012001 2011-12-26 2012-01-01
4 2012 1 2 12000 2012011 2012-01-02 2012-01-02
5 2012 1 3 11000 2012021 2012-01-09 2012-01-09
What about this ?
df['date_start'] = pd.to_datetime(df.week.astype(str)+
df.year.astype(str).add('-1') ,format='%V%G-%u')
This will give date_start as the date of the Monday of the week of interest.
(Note that there is a shift with your current date_start, you might want to add a 1 week tmimedelta to compensate for it.
I have read a couple of similar post regarding the issue before, but none of the solutions worked for me. so I got the followed csv :
Score date term
0 72 3 Feb · 1
1 47 1 Feb · 1
2 119 6 Feb · 1
8 101 7 hrs · 1
9 536 11 min · 1
10 53 2 hrs · 1
11 20 11 Feb · 3
3 15 1 hrs · 2
4 33 7 Feb · 1
5 153 4 Feb · 3
6 34 3 min · 2
7 26 3 Feb · 3
I want to sort the csv by date. What's the easiest way to do that ?
You can create 2 helper columns - one for datetimes created by to_datetime and second for timedeltas created by to_timedelta, only necessary format HH:MM:SS, so added Series.replace by regexes, so last is possible sorting by 2 columns by DataFrame.sort_values:
df['date1'] = pd.to_datetime(df['date'], format='%d %b', errors='coerce')
times = df['date'].replace({'(\d+)\s+min': '00:\\1:00',
'\s+hrs': ':00:00'}, regex=True)
df['times'] = pd.to_timedelta(times, errors='coerce')
df = df.sort_values(['times','date1'])
print (df)
Score date term date1 times
6 34 3 min 2 NaT 00:03:00
9 536 11 min 1 NaT 00:11:00
3 15 1 hrs 2 NaT 01:00:00
10 53 2 hrs 1 NaT 02:00:00
8 101 7 hrs 1 NaT 07:00:00
1 47 1 Feb 1 1900-02-01 NaT
0 72 3 Feb 1 1900-02-03 NaT
7 26 3 Feb 3 1900-02-03 NaT
5 153 4 Feb 3 1900-02-04 NaT
2 119 6 Feb 1 1900-02-06 NaT
4 33 7 Feb 1 1900-02-07 NaT
11 20 11 Feb 3 1900-02-11 NaT
I have a pandas column like this :
yrmnt
--------
2015 03
2015 03
2013 08
2015 08
2014 09
2015 10
2016 02
2015 11
2015 11
2015 11
2017 02
How to fetch lowest year month combination :2013 08 and highest : 2017 02
And find the difference in months between these two, ie 40
You can connvert column to_datetime and then find indices by max and min values by idxmax and
idxmin:
a = pd.to_datetime(df['yrmnt'], format='%Y %m')
print (a)
0 2015-03-01
1 2015-03-01
2 2013-08-01
3 2015-08-01
4 2014-09-01
5 2015-10-01
6 2016-02-01
7 2015-11-01
8 2015-11-01
9 2015-11-01
10 2017-02-01
Name: yrmnt, dtype: datetime64[ns]
print (df.loc[a.idxmax(), 'yrmnt'])
2017 02
print (df.loc[a.idxmin(), 'yrmnt'])
2013 08
Difference in months:
b = a.dt.to_period('M')
d = b.max() - b.min()
print (d)
42
Another solution working only with month period created by Series.dt.to_period:
b = pd.to_datetime(df['yrmnt'], format='%Y %m').dt.to_period('M')
print (b)
0 2015-03
1 2015-03
2 2013-08
3 2015-08
4 2014-09
5 2015-10
6 2016-02
7 2015-11
8 2015-11
9 2015-11
10 2017-02
Name: yrmnt, dtype: object
Then convert to custom format by Period.strftime minimal and maximal values:
min_d = b.min().strftime('%Y %m')
print (min_d)
2013 08
max_d = b.max().strftime('%Y %m')
print (max_d)
2017 02
And subtract for difference:
d = b.max() - b.min()
print (d)
42
I have some data that looks like:
key DATE - DAY DATE - MONTH DATE - YEAR GMT HRS GMT MINUTES
1 2 29 2 2016 2 2
2 3 29 2 2016 2 2
3 4 29 2 2016 2 2
4 5 29 2 2016 2 2
5 6 29 2 2016 2 2
6 7 29 2 2016 2 2
7 8 29 2 2016 2 3
8 9 29 2 2016 2 3
9 10 29 2 2016 2 3
GMT SECONDS
1 54
2 55
3 56
4 57
5 58
6 59
7 0
8 1
9 2
At first the data was type float and the year was in format 16 so I did:
t['DATE - MONTH'] = t['DATE - MONTH'].astype(int)
t['DATE - YEAR'] = t['DATE - YEAR'].astype(int)
t['DATE - YEAR'] = t['DATE - YEAR']+2000
t['DATE - DAY'] = t['DATE - DAY'].astype(int)
^Note I was also confused why when using an index number rather than the column name you only work on what seems to be a temp table ie you can print the desired result but it didnt change the data frame.
Then I tried two methods:
t['Date'] = pd.to_datetime(dict(year=t['DATE - YEAR'], month = t['DATE - MONTH'], day = t['DATE - DAY']))
t['Date'] = pd.to_datetime((t['DATE - YEAR']*10000+t['DATE - MONTH']*100+t['DATE - DAY']).apply(str),format='%Y%m%d')
Both return:
ValueError: cannot assemble the datetimes: time data 20000000 does not match format '%Y%m%d' (match)
I'd like to create a date column (and then after use a similar logic for a datetime column with the additional 3 columns).
What is the problem?
EDIT: I had bad data and added errors='coerce' to handle those rows
First rename all columns, filter by values of dict and use to_datetime:
Assembling a datetime from multiple columns of a DataFrame. The keys can be common abbreviations like ['year', 'month', 'day', 'minute', 'second', 'ms', 'us', 'ns']) or plurals of the same.
d = {'DATE - YEAR':'year','DATE - MONTH':'month','DATE - DAY':'day',
'GMT HRS':'hour','GMT MINUTES':'minute','GMT SECONDS':'second'}
df['datetime'] = pd.to_datetime(df.rename(columns=d)[list(d.values())])
print (df)
key DATE - DAY DATE - MONTH DATE - YEAR GMT HRS GMT MINUTES \
1 2 29 2 2016 2 2
2 3 29 2 2016 2 2
3 4 29 2 2016 2 2
4 5 29 2 2016 2 2
5 6 29 2 2016 2 2
6 7 29 2 2016 2 2
7 8 29 2 2016 2 3
8 9 29 2 2016 2 3
9 10 29 2 2016 2 3
GMT SECONDS datetime
1 54 2016-02-29 02:02:54
2 55 2016-02-29 02:02:55
3 56 2016-02-29 02:02:56
4 57 2016-02-29 02:02:57
5 58 2016-02-29 02:02:58
6 59 2016-02-29 02:02:59
7 0 2016-02-29 02:03:00
8 1 2016-02-29 02:03:01
9 2 2016-02-29 02:03:02
Detail:
print (df.rename(columns=d)[list(d.values())])
day month second year minute hour
1 29 2 54 2016 2 2
2 29 2 55 2016 2 2
3 29 2 56 2016 2 2
4 29 2 57 2016 2 2
5 29 2 58 2016 2 2
6 29 2 59 2016 2 2
7 29 2 0 2016 3 2
8 29 2 1 2016 3 2
9 29 2 2 2016 3 2