Why is this lambda operation not working? - python

I want to take any values in my dataframe that are shown as 'less than' and report them as numbers half of the less-than value.
e.g. <1 becomes 0.5, <0.5 becomes 0.25, <5 becomes 2.5 etc.
ordinary numbers and text should be unchanged.
I have the following lambda function to apply to my dataframe that I thought was working but it isn't:
df_no_less_thans= df.apply(lambda x: x if str(x[0])!='<' else float(x[1:])/2)
I am still getting '<' values in the new df, no error messages.
What have I done wrong?
df=pd.DataFrame()
df['Cu']=[3.7612,1.3693, 2.7502,1.407,4.2066,6.4409,6.8136,"<0.05","<0.05",0.94,0.07,1.82,2.63,1.36,0.78]
df.apply(lambda x: x if str(x)[0]!='<' else float(str(x)[1:])/2)
df
gives
Cu
0 3.7612
1 1.3693
2 2.7502
3 1.407
4 4.2066
5 6.4409
6 6.8136
7 <0.05
8 <0.05
9 0.94
10 0.07
11 1.82
12 2.63
13 1.36
14 0.7 ```


Your code won't work with non-strings like integers or floats since you cannot index them without converting them to a string. You can explicitly cast everything to string and perform your indexing
You would also want to have a check for empty strings before you perform the lambda operation
#Explicitly cast to string and perform the indexing
func = lambda x: x if str(x)[0]!='<' else float(str(x)[1:])/2
li = ['<1', '<0.5', '<5', 1, 'hello', 4.0, '']
#Filter out empty strings
print([func(item) for item in li if item])
The output will be
[0.5, 0.25, 2.5, 1, 'hello', 4.0]


The method apply has an axis argument. By default, axis=0, which means that your lambda function is applied successively to each column in the dataframe. In your case, the lambda function is applied to the column 'Cu', meaning that the argument x is actually a column and str(x)[0] is not what you think.
You should use applymap instead, to apply the lambda function element-wise:
df.applymap(lambda x: x if str(x)[0] != '<' else float(str(x)[1:])/2)


I think you need apply lambda function only for Cu column, so correct solution is use Series.apply:
df['Cu'] = df['Cu'].apply(lambda x: x if str(x)[0]!='<' else float(str(x)[1:])/2)
print (df)
Cu
0 3.7612
1 1.3693
2 2.7502
3 1.4070
4 4.2066
5 6.4409
6 6.8136
7 0.0250
8 0.0250
9 0.9400
10 0.0700
11 1.8200
12 2.6300
13 1.3600
14 0.7800
If need apply function for all columns use IanS solution.


Here is how it works:
import pandas as pd
df=pd.DataFrame()
df['Cu']=[3.7612,1.3693, 2.7502,1.407,4.2066,6.4409,6.8136,"<0.05","<0.05",0.94,0.07,1.82,2.63,1.36,0.78]
df['Cu'] = df.apply(lambda x: x if not isinstance(x[0],str) else float(x[0][1:])/2, axis=1, raw=True)
print(df)
result:
Cu
0 3.7612
1 1.3693
2 2.7502
3 1.407
4 4.2066
5 6.4409
6 6.8136
7 0.025
8 0.025
9 0.94
10 0.07
11 1.82
12 2.63
13 1.36
14 0.78


In your question you say
e.g. <1 becomes 0.5, <0.5 becomes 0.25, <5 becomes 2.5 etc. ordinary numbers and text should be unchanged.
Now in the example you have given you only have the first two types of data: strings like <1 and floats, but you seem to want to be able to retain any kind of other text, too. However I see mixing different dtypes in one column as a bad dataframe layout that will only cause trouble in the future.
If, for example, you had some text hello in your column, a simple operation like:
df['Cu'] * 2
# [...]
# 6 13.6272
# 7 hellohello
# 8 0.05
# 9 1.88
# [...]
# Name: Cu, dtype: object
This is most likely not what you want.
Now I don't know what other kinds of text you have, but for the examples given I would recommend normalizing the dtypes first: For that we create a new column df['less_than'] from the "uncertainty information":
import pandas as pd
df=pd.DataFrame()
df['Cu']=[3.7612,1.3693, 2.7502,1.407,4.2066,6.4409,6.8136,"<0.05","<0.05",0.94,0.07,1.82,2.63,1.36,0.78]
df['less_than'] = df['Cu'].str.startswith('<', False)
df.loc[df['less_than'], 'Cu'] = df.loc[df['less_than'], 'Cu'].str.slice(1)
df['Cu'] = df['Cu'].astype(float)
# Cu less_than
# 0 3.7612 False
# 1 1.3693 False
# 2 2.7502 False
# 3 1.4070 False
# 4 4.2066 False
# 5 6.4409 False
# 6 6.8136 False
# 7 0.0500 True
# 8 0.0500 True
# 9 0.9400 False
# 10 0.0700 False
# 11 1.8200 False
# 12 2.6300 False
# 13 1.3600 False
# 14 0.7800 False
This enables us to do treat the entire column df['Cu'] the same, and making your "<1 becomes 0.5" operations a simple one-liner:
df.loc[df['less_than'], 'Cu'] /= 2

Related

I need to create a dataframe were values reference previous rows

I am just starting to use python and im trying to learn some of the general things about it. As I was playing around with it I wanted to see if I could make a dataframe that shows a starting number which is compounded by a return. Sorry if this description doesnt make much sense but I basically want a dataframe x long that shows me:
number*(return)^(row number) in each row
so for example say number is 10 and the return is 10% so i would like for the dataframe to give me the series
1 11
2 12.1
3 13.3
4 14.6
5 ...
6 ...
Thanks so much in advanced!

Let us try
import numpy as np
val = 10
det = 0.1
n = 4
out = 10*((1+det)**np.arange(n))
s = pd.Series(out)
s
Out[426]:
0 10.00
1 11.00
2 12.10
3 13.31
dtype: float64
Notice here I am using the index from 0 , since 1.1**0 will yield the original value

I think this does what you want:
df = pd.DataFrame({'returns': [x for x in range(1, 10)]})
df.index = df.index + 1
df.returns = df.returns.apply(lambda x: (10 * (1.1**x)))
print(df)
Out:
returns
1 11.000000
2 12.100000
3 13.310000
4 14.641000
5 16.105100
6 17.715610
7 19.487171
8 21.435888
9 23.579477

lambda function referencing a column value not specified in function

I have a situation where I want to use the results of a groupby in my training set to fill in results for my test set.
I don't think there's a straight forward way to do this in pandas, so I'm trying use the apply method on the column in my test set.
MY SITUATION:
I want to use the average values from my MSZoning column to infer the missing value for my LotFrontage column.
If I use the groupby method on my training set I get this:
train.groupby('MSZoning')['LotFrontage'].agg(['mean', 'count'])
giving.....
Now, I want to use these values to impute missing values on my test set, so I can't just use the transform method.
Instead, I created a function that I wanted to pass into the apply method, which can be seen here:
def fill_MSZoning(row):
if row['MSZoning'] == 'C':
return 69.7
elif row['MSZoning'] == 'FV':
return 59.49
elif row['MSZoning'] == 'RH':
return 58.92
elif row['MSZoning'] == 'RL':
return 74.68
else:
return 52.4
I call the function like this:
test['LotFrontage'] = test.apply(lambda x: x.fillna(fill_MSZoning), axis=1)
Now, the results for the LotFrontage column are the same as the Id column, even though I didn't specify this.
Any idea what is happening?

you can do it like this
import pandas as pd
import numpy as np
## creating dummy data
np.random.seed(100)
raw = {
"group": np.random.choice("A B C".split(), 10),
"value": [np.nan if np.random.rand()>0.8 else np.random.choice(100) for _ in range(10)]
}
df = pd.DataFrame(raw)
display(df)
## calculate mean
means = df.groupby("group").mean()
display(means)
Fill With Group Mean
## fill with mean value
def fill_group_mean(x):
group_mean = means["value"].loc[x["group"].max()]
return x["value"].mask(x["value"].isna(), group_mean)
r= df.groupby("group").apply(fill_group_mean)
r.reset_index(level=0)
Output
group value
0 A NaN
1 A 24.0
2 A 60.0
3 C 9.0
4 C 2.0
5 A NaN
6 C NaN
7 B 83.0
8 C 91.0
9 C 7.0
group value
0 A 42.00
1 A 24.00
2 A 60.00
5 A 42.00
7 B 83.00
3 C 9.00
4 C 2.00
6 C 27.25
8 C 91.00
9 C 7.00

Search for value within a range in a pandas dataframe?

I am attempting to search for matching values within a range within a given uncertainty in a pandas dataframe. For instance, if I have a dataframe:
A B C
0 12 12.6 111.20
1 14 23.4 112.20
2 16 45.6 112.30
3 18 56.6 112.40
4 27 34.5 121.60
5 29 65.2 223.23
6 34 45.5 654.50
7 44 65.6 343.50
How can I search for a value that matches 112.6 +/-0.4 without having to create a long and difficult criteria like:
TargetVal_Max= 112.6+0.4
TargetVal_Min= 112.6-0.4
Basically, I want to create a "buffer window" that allows for all values matching a window to be returned back. I have uncertainties package, but have yet to get it working like this.
Optimally, I'd like to be able to return all index values that match a value in both C and B within a given error range.
Edit
As pointed out by #MaxU, the np.isclose f(x) works very well if you know the exact number. But is it possible to match a list of values, such that if I had a second dataframe and wanted to see if the values in C from one matched the values of C (second dataframe) within a tolerance? I have attempted to get them into a list and do it this way, but I am getting problems when attempting to do it for more than a single value at a time.
TEST= Dataframe_2["C"]
HopesNdreams = sample[sample["C"].apply(np.isclose,b=TEST, atol=1.0)]
Edit 2
I found through trying a couple of different work arounds that I can just do:
TEST1= Dataframe_2["C"].tolist
for i in TEST1:
HopesNdreams= sample[sample["C"].apply(np.isclose,b=i, atol=1.0)]
And this returns the hits for the given column. Using the logic set forth in the first answer, I think this will work very well for what I need it to. Are there any hangups that I don't see with this method?
Cheers and thanks for the help!

IIUC you can use np.isclose() function:
In [180]: df[['B','C']].apply(np.isclose, b=112.6, atol=0.4)
Out[180]:
B C
0 False False
1 False True
2 False True
3 False True
4 False False
5 False False
6 False False
7 False False
In [181]: df[['B','C']].apply(np.isclose, b=112.6, atol=0.4).any(1)
Out[181]:
0 False
1 True
2 True
3 True
4 False
5 False
6 False
7 False
dtype: bool
In [182]: df[df[['B','C']].apply(np.isclose, b=112.6, atol=0.4).any(1)]
Out[182]:
A B C
1 14 23.4 112.2
2 16 45.6 112.3
3 18 56.6 112.4

Use Series.between():
df['C'].between(112.6 + .4, 112.6 - .4)

Python testing true/false condition on data frame column and returning output in new column

I'm very new to coding in Python so I'm trying to get to grips with some basics - any input is appreciated.
I have a list of weekly dates, and am trying to run an 'if' statement on the days, i.e. if the day number is less than 7, create a column with a factor x, or else create a factor y - as in the table below:
week day check factor
0 2017-01-08 8 False x
1 2017-01-15 15 False x
2 2017-01-22 22 False x
3 2017-01-29 29 False x
4 2017-02-05 5 True y
I tried the code below:
if df['day'] <7 :
factor=weeks['day']/7
else:
....
and got an error:
ValueError: The truth value of a Series is ambiguous
which I have looked into, and understand that the code above is attempting to test the whole column and hence there can not be an unambiguous true/false response. I have seen some comments about all/any, but these also do not give me the response I'm looking for. Is there a way of testing each item in a column and returning a different output depending on the value?

If needed convert the 'week' column datetime dtype using to_datetime then you can just compare the day attribute using dt.day and use this as the condition for np.where:
In [47]:
df['week'] = pd.to_datetime(df['week'])
df['factor'] = np.where(df['week'].dt.day < 7, 'y', 'x')
df
Out[47]:
week day check factor
0 2017-01-08 8 False x
1 2017-01-15 15 False x
2 2017-01-22 22 False x
3 2017-01-29 29 False x
4 2017-02-05 5 True y

You can use apply.
This should work for your use case:
df['factor'] = df.apply(lambda row: "y" if row['day'] < 7 else "x", axis=1)
This will iterate over each row (axis=1), and check the day value. If it is less than 7, the value will be 'y', else it will be 'n'.
This outputs (for example):
check day week factor
0 False 8 2017-01-08 x
1 True 5 2017-02-05 y
Also, since you already have a "day" column, you can just work with it directly and avoid an apply:
msk = df['day'] < 7 # Series of True/False values based on condition
df[factor] = msk.replace([True, False], ['y', 'x']) #convert True/False to y/x

Pandas dataframe total row

I have a dataframe, something like:
foo bar qux
0 a 1 3.14
1 b 3 2.72
2 c 2 1.62
3 d 9 1.41
4 e 3 0.58
and I would like to add a 'total' row to the end of dataframe:
foo bar qux
0 a 1 3.14
1 b 3 2.72
2 c 2 1.62
3 d 9 1.41
4 e 3 0.58
5 total 18 9.47
I've tried to use the sum command but I end up with a Series, which although I can convert back to a Dataframe, doesn't maintain the data types:
tot_row = pd.DataFrame(df.sum()).T
tot_row['foo'] = 'tot'
tot_row.dtypes:
foo object
bar object
qux object
I would like to maintain the data types from the original data frame as I need to apply other operations to the total row, something like:
baz = 2*tot_row['qux'] + 3*tot_row['bar']

Update June 2022
pd.append is now deprecated. You could use pd.concat instead but it's probably easier to use df.loc['Total'] = df.sum(numeric_only=True), as Kevin Zhu commented. Or, better still, don't modify the data frame in place and keep your data separate from your summary statistics!
Append a totals row with
df.append(df.sum(numeric_only=True), ignore_index=True)
The conversion is necessary only if you have a column of strings or objects.
It's a bit of a fragile solution so I'd recommend sticking to operations on the dataframe, though. eg.
baz = 2*df['qux'].sum() + 3*df['bar'].sum()

df.loc["Total"] = df.sum()
works for me and I find it easier to remember. Am I missing something?
Probably wasn't possible in earlier versions.
I'd actually like to add the total row only temporarily though.
Adding it permanently is good for display but makes it a hassle in further calculations.
Just found
df.append(df.sum().rename('Total'))
This prints what I want in a Jupyter notebook and appears to leave the df itself untouched.

New Method
To get both row and column total:
import numpy as np
import pandas as pd
df = pd.DataFrame({'a': [10,20],'b':[100,200],'c': ['a','b']})
df.loc['Column_Total']= df.sum(numeric_only=True, axis=0)
df.loc[:,'Row_Total'] = df.sum(numeric_only=True, axis=1)
print(df)
a b c Row_Total
0 10.0 100.0 a 110.0
1 20.0 200.0 b 220.0
Column_Total 30.0 300.0 NaN 330.0

Use DataFrame.pivot_table with margins=True:
import pandas as pd
data = [('a',1,3.14),('b',3,2.72),('c',2,1.62),('d',9,1.41),('e',3,.58)]
df = pd.DataFrame(data, columns=('foo', 'bar', 'qux'))
Original df:
foo bar qux
0 a 1 3.14
1 b 3 2.72
2 c 2 1.62
3 d 9 1.41
4 e 3 0.58
Since pivot_table requires some sort of grouping (without the index argument, it'll raise a ValueError: No group keys passed!), and your original index is vacuous, we'll use the foo column:
df.pivot_table(index='foo',
margins=True,
margins_name='total', # defaults to 'All'
aggfunc=sum)
Voilà!
bar qux
foo
a 1 3.14
b 3 2.72
c 2 1.62
d 9 1.41
e 3 0.58
total 18 9.47

Alternative way (verified on Pandas 0.18.1):
import numpy as np
total = df.apply(np.sum)
total['foo'] = 'tot'
df.append(pd.DataFrame(total.values, index=total.keys()).T, ignore_index=True)
Result:
foo bar qux
0 a 1 3.14
1 b 3 2.72
2 c 2 1.62
3 d 9 1.41
4 e 3 0.58
5 tot 18 9.47

Building on JMZ answer
df.append(df.sum(numeric_only=True), ignore_index=True)
if you want to continue using your current index you can name the sum series using .rename() as follows:
df.append(df.sum().rename('Total'))
This will add a row at the bottom of the table.

This is the way that I do it, by transposing and using the assign method in combination with a lambda function. It makes it simple for me.
df.T.assign(GrandTotal = lambda x: x.sum(axis=1)).T

Building on answer from Matthias Kauer.
To add row total:
df.loc["Row_Total"] = df.sum()
To add column total,
df.loc[:,"Column_Total"] = df.sum(axis=1)

New method [September 2022]
TL;DR:
Just use
df.style.concat(df.agg(['sum']).style)
for a solution that won't change you dataframe, works even if you have an "sum" in your index, and can be styled!
Explanation
In pandas 1.5.0, a new method named .style.concat() gives you the ability to display several dataframes together. This is a good way to show the total (or any other statistics), because it is not changing the original dataframe, and works even if you have an index named "sum" in your original dataframe.
For example:
import pandas as pd
df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], columns=['A', 'B', 'C'])
df.style.concat(df.agg(['sum']).style)
and it will return a formatted table that is visible in jupyter as this:
Styling
with a little longer code, you can even make the last row look different:
df.style.concat(
df.agg(['sum']).style
.set_properties(**{'background-color': 'yellow'})
)
to get:
see other ways to style (such as bold font, or table lines) in the docs

Following helped for me to add a column total and row total to a dataframe.
Assume dft1 is your original dataframe... now add a column total and row total with the following steps.
from io import StringIO
import pandas as pd
#create dataframe string
dfstr = StringIO(u"""
a;b;c
1;1;1
2;2;2
3;3;3
4;4;4
5;5;5
""")
#create dataframe dft1 from string
dft1 = pd.read_csv(dfstr, sep=";")
## add a column total to dft1
dft1['Total'] = dft1.sum(axis=1)
## add a row total to dft1 with the following steps
sum_row = dft1.sum(axis=0) #get sum_row first
dft1_sum=pd.DataFrame(data=sum_row).T #change it to a dataframe
dft1_sum=dft1_sum.reindex(columns=dft1.columns) #line up the col index to dft1
dft1_sum.index = ['row_total'] #change row index to row_total
dft1.append(dft1_sum) # append the row to dft1

Actually all proposed solutions render the original DataFrame unusable for any further analysis and can invalidate following computations, which will be easy to overlook and could lead to false results.
This is because you add a row to the data, which Pandas cannot differentiate from an additional row of data.
Example:
import pandas as pd
data = [1, 5, 6, 8, 9]
df = pd.DataFrame(data)
df
df.describe()
yields
0
0
1
1
5
2
6
3
8
4
9
0
count
5
mean
5.8
std
3.11448
min
1
25%
5
50%
6
75%
8
max
9
After
df.loc['Totals']= df.sum(numeric_only=True, axis=0)
the dataframe looks like this
0
0
1
1
5
2
6
3
8
4
9
Totals
29
This looks nice, but the new row is treated as if it was an additional data item, so df.describe will produce false results:
0
count
6
mean
9.66667
std
9.87252
min
1
25%
5.25
50%
7
75%
8.75
max
29
So: Watch out! and apply this only after doing all other analyses of the data or work on a copy of the DataFrame!

When the "totals" need to be added to an index column:
totals = pd.DataFrame(df.sum(numeric_only=True)).transpose().set_index(pd.Index({"totals"}))
df.append(totals)
e.g.
(Pdb) df
count min bytes max bytes mean bytes std bytes sum bytes
row_0 837200 67412.0 368733992.0 2.518989e+07 5.122836e+07 2.108898e+13
row_1 299000 85380.0 692782132.0 2.845055e+08 2.026823e+08 8.506713e+13
row_2 837200 67412.0 379484173.0 8.706825e+07 1.071484e+08 7.289354e+13
row_3 239200 85392.0 328063972.0 9.870446e+07 1.016989e+08 2.361011e+13
row_4 59800 67292.0 383487021.0 1.841879e+08 1.567605e+08 1.101444e+13
row_5 717600 112309.0 379483824.0 9.687554e+07 1.103574e+08 6.951789e+13
row_6 119600 664144.0 358486985.0 1.611637e+08 1.171889e+08 1.927518e+13
row_7 478400 67300.0 593141462.0 2.824301e+08 1.446283e+08 1.351146e+14
row_8 358800 215002028.0 327493141.0 2.861329e+08 1.545693e+07 1.026645e+14
row_9 358800 202248016.0 321657935.0 2.684668e+08 1.865470e+07 9.632590e+13
(Pdb) totals = pd.DataFrame(df.sum(numeric_only=True)).transpose()
(Pdb) totals
count min bytes max bytes mean bytes std bytes sum bytes
0 4305600.0 418466685.0 4.132815e+09 1.774725e+09 1.025805e+09 6.365722e+14
(Pdb) totals = pd.DataFrame(df.sum(numeric_only=True)).transpose().set_index(pd.Index({"totals"}))
(Pdb) totals
count min bytes max bytes mean bytes std bytes sum bytes
totals 4305600.0 418466685.0 4.132815e+09 1.774725e+09 1.025805e+09 6.365722e+14
(Pdb) df.append(totals)
count min bytes max bytes mean bytes std bytes sum bytes
row_0 837200.0 67412.0 3.687340e+08 2.518989e+07 5.122836e+07 2.108898e+13
row_1 299000.0 85380.0 6.927821e+08 2.845055e+08 2.026823e+08 8.506713e+13
row_2 837200.0 67412.0 3.794842e+08 8.706825e+07 1.071484e+08 7.289354e+13
row_3 239200.0 85392.0 3.280640e+08 9.870446e+07 1.016989e+08 2.361011e+13
row_4 59800.0 67292.0 3.834870e+08 1.841879e+08 1.567605e+08 1.101444e+13
row_5 717600.0 112309.0 3.794838e+08 9.687554e+07 1.103574e+08 6.951789e+13
row_6 119600.0 664144.0 3.584870e+08 1.611637e+08 1.171889e+08 1.927518e+13
row_7 478400.0 67300.0 5.931415e+08 2.824301e+08 1.446283e+08 1.351146e+14
row_8 358800.0 215002028.0 3.274931e+08 2.861329e+08 1.545693e+07 1.026645e+14
row_9 358800.0 202248016.0 3.216579e+08 2.684668e+08 1.865470e+07 9.632590e+13
totals 4305600.0 418466685.0 4.132815e+09 1.774725e+09 1.025805e+09 6.365722e+14

Since i generally want to do this at the very end as to avoid breaking the integrity of the dataframe (right before printing). I created a summary_rows_cols method which returns a printable dataframe:
def summary_rows_cols(df: pd.DataFrame,
column_sum: bool = False,
column_avg: bool = False,
column_median: bool = False,
row_sum: bool = False,
row_avg: bool = False,
row_median: bool = False
) -> pd.DataFrame:
ret = df.copy()
if column_sum: ret.loc['Sum'] = df.sum(numeric_only=True, axis=0)
if column_avg: ret.loc['Avg'] = df.mean(numeric_only=True, axis=0)
if column_median: ret.loc['Median'] = df.median(numeric_only=True, axis=0)
if row_sum: ret.loc[:, 'Sum'] = df.sum(numeric_only=True, axis=1)
if row_median: ret.loc[:, 'Avg'] = df.mean(numeric_only=True, axis=1)
if row_avg: ret.loc[:, 'Median'] = df.median(numeric_only=True, axis=1)
ret.fillna('-', inplace=True)
return ret
This allows me to enter a generic (numeric) df and get a summarized output such as:
a b c Sum Median
0 1 4 7 12 4
1 2 5 8 15 5
2 3 6 9 18 6
Sum 6 15 24 - -
from:
data = {
'a': [1, 2, 3],
'b': [4, 5, 6],
'c': [7, 8, 9]
}
df = pd.DataFrame(data)
printable = summary_rows_cols(df, row_sum=True, column_sum=True, row_median=True)

Categories

Resources