Develop Reference

Is .isin() faster than .query()

Question:
Hi,
When searching for methods to make a selection of a dataframe (being relatively unexperienced with Pandas), I had the following question:
What is faster for large datasets - .isin() or .query()?
Query is somewhat more intuitive to read, so my preferred approach due to my line of work. However, testing it on a very small example dataset, query seems to be much slower.
Is there anyone who has tested this properly before? If so, what were the outcomes? I searched the web, but could not find another post on this.
See the sample code below, which works for Python 3.8.5.
Thanks a lot in advance for your help!
Code:
# Packages
import pandas as pd
import timeit
import numpy as np
# Create dataframe
df = pd.DataFrame({'name': ['Foo', 'Bar', 'Faz'],
'owner': ['Canyon', 'Endurace', 'Bike']},
index=['Frame', 'Type', 'Kind'])
# Show dataframe
df
# Create filter
selection = ['Canyon']
# Filter dataframe using 'isin' (type 1)
df_filtered = df[df['owner'].isin(selection)]
%timeit df_filtered = df[df['owner'].isin(selection)]
213 µs ± 14 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
# Filter dataframe using 'isin' (type 2)
df[np.isin(df['owner'].values, selection)]
%timeit df_filtered = df[np.isin(df['owner'].values, selection)]
128 µs ± 3.11 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# Filter dataframe using 'query'
df_filtered = df.query("owner in #selection")
%timeit df_filtered = df.query("owner in #selection")
1.15 ms ± 9.35 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

The best test in real data, here fast comparison for 3k, 300k,3M rows with this sample data:
selection = ['Hedge']
df = pd.concat([df] * 1000, ignore_index=True)
In [139]: %timeit df[df['owner'].isin(selection)]
449 µs ± 58 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [140]: %timeit df.query("owner in #selection")
1.57 ms ± 33.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
df = pd.concat([df] * 100000, ignore_index=True)
In [142]: %timeit df[df['owner'].isin(selection)]
8.25 ms ± 66.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [143]: %timeit df.query("owner in #selection")
13 ms ± 1.05 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
df = pd.concat([df] * 1000000, ignore_index=True)
In [145]: %timeit df[df['owner'].isin(selection)]
94.5 ms ± 9.28 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [146]: %timeit df.query("owner in #selection")
112 ms ± 499 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
If check docs:
DataFrame.query() using numexpr is slightly faster than Python for large frames
Conclusion - The best test in real data, because depends of number of rows, number of matched values and also by length of list selection.

A perfplot over some generated data:
Assuming some hypothetical data, as well as a proportionally increasing selection size (10% of frame size).
Sample data for n=10:
df:
name owner
0 Constant JoVMq
1 Constant jiKNB
2 Constant WEqhm
3 Constant pXNqB
4 Constant SnlbV
5 Constant Euwsj
6 Constant QPPbs
7 Constant Nqofa
8 Constant qeUKP
9 Constant ZBFce
Selection:
['ZBFce']
Performance reflects the docs. At smaller frames the overhead of query is significant over isin However, at frames around 200k rows the performance is comparable to isin and at frames around 10m rows query starts to become more performant.
I agree with #jezrael that, this is, as with most pandas runtime problems, very data dependent, and the best test would be to test on real datasets for a given use case and make a decision based on that.
Edit: Included #AlexanderVolkovsky's suggestion to convert selection to a set and use apply + in:
Perfplot Code:
import string
import numpy as np
import pandas as pd
import perfplot
charset = list(string.ascii_letters)
np.random.seed(5)
def gen_data(n):
df = pd.DataFrame({'name': 'Constant',
'owner': [''.join(np.random.choice(charset, 5))
for _ in range(n)]})
selection = df['owner'].sample(frac=.1).tolist()
return df, selection, set(selection)
def test_isin(params):
df, selection, _ = params
return df[df['owner'].isin(selection)]
def test_query(params):
df, selection, _ = params
return df.query("owner in #selection")
def test_apply_over_set(params):
df, _, set_selection = params
return df[df['owner'].apply(lambda x: x in set_selection)]
if __name__ == '__main__':
out = perfplot.bench(
setup=gen_data,
kernels=[
test_isin,
test_query,
test_apply_over_set
],
labels=[
'test_isin',
'test_query',
'test_apply_over_set'
],
n_range=[2 ** k for k in range(25)],
equality_check=None
)
out.save('perfplot_results.png', transparent=False)

How to round calculations with pandas

I know how to simply round the column in pandas (link), however, my problem is how can I round and do calculation at the same time in pandas.
df['age_new'] = df['age'].apply(lambda x: round(x['age'] * 0.024319744084, 0.000000000001))
TypeError: 'float' object is not subscriptable
Is there any way to do this?

.apply is not vectorized.
When using .apply on a pandas.Series, like 'age', the lambda variable, x is the 'age' column, so the correct syntax is round(x * 0.0243, 4)
The ndigits parameter of round, requires an int, not a float.
It is faster to use vectorized methods, like .mul, and then .round.
In this case, with 1000 rows, the vectorized method is 4 times faster than using .apply.
import pandas as pd
import numpy as np
# test data
np.random.seed(365)
df = pd.DataFrame({'age': np.random.randint(110, size=(1000))})
%%timeit
df.age.mul(0.024319744084).round(5)
[out]:
212 µs ± 3.86 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%%timeit
(df['age'] * 0.024319744084).round(5)
[out]:
211 µs ± 9.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%%timeit
df.age.apply(lambda x: round(x * 0.024319744084, 5))
[out]:
845 µs ± 20.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

There's two problems:
x['age'] inside the brackets doesn't need ['age'] as you already apply to the column age (that's why you get the error)
round takes an int as second argument.
Try
df['age_new'] = df['age'].apply(lambda x: round(x * 0.024319744084, 5))
(5 is just an example.)

Pandas startswith operation between two columns

I have a pandas dataframe with two columns, where I need to check where the value at each row of column A is a string that starts with the value of the corresponding row at column B or viceversa.
It seems that the Series method .str.startswith cannot deal with vectorized input, so I needed to zip over the two columns in a list comprehension and create a new pd.Series with the same index as any of the two columns.
I would like this to be a vectorized operation with the .str accessor available to operate on iterables, but something like this returns NaN:
df = pd.DataFrame(data={'a':['x','yy'], 'b':['xyz','uvw']})
df['a'].str.startswith(df['b'])
while my working solution is the following:
pd.Series(index=df.index, data=[a.startswith(b) or b.startswith(a) for a,b in zip(df['a'],df['b'])])
I suspect that there may be a better way to tackle this issue as it also would benefit all string methods on series.
Is there any more beautiful or efficient method to do this?

One idea is use np.vecorize, but because working with strings performance is only a bit better like your solution:
def fun (a,b):
return a.startswith(b) or b.startswith(a)
f = np.vectorize(fun)
a = pd.Series(f(df['a'],df['b']), index=df.index)
print (a)
0 True
1 False
dtype: bool
df = pd.DataFrame(data={'a':['x','yy'], 'b':['xyz','uvw']})
df = pd.concat([df] * 10000, ignore_index=True)
In [132]: %timeit pd.Series(index=df.index, data=[a.startswith(b) or b.startswith(a) for a,b in df[['a', 'b']].to_numpy()])
42.3 ms ± 516 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [133]: %timeit pd.Series(f(df['a'],df['b']), index=df.index)
9.81 ms ± 119 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [134]: %timeit pd.Series(index=df.index, data=[a.startswith(b) or b.startswith(a) for a,b in zip(df['a'],df['b'])])
14.1 ms ± 262 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
#sammywemmy solution
In [135]: %timeit pd.Series([any((a.startswith(b), b.startswith(a))) for a, b in df.to_numpy()], index=df.index)
46.3 ms ± 683 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Use a function instead of string in Pandas Groupby Agg

When aggregating data in Pandas I am able to return strings like "count", "sum", "mean", etc to aggregate data. Are there functions I can use instead of strings that would provide equivalent behavior. For example, if I try to use pd.Series.Count instead of count, the runtime takes a sizable hit.
import pandas as pd
import numpy as np
n = 10000000
df_nan = pd.DataFrame({"a": np.random.randint(0, 100, n*2),
"b": np.linspace(0, 100, n).tolist() + [None]*n})
%timeit df_nan.groupby("a").agg({"b": pd.Series.count})
1.63 s ± 28 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit df_nan.groupby("a").agg({"b": "count"})
479 ms ± 18.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Any idea what function I could return instead?

Why creating new column on a Pandas dataframe with not sorted index is slow

My goal is to perform some basic calculation with the first occurring row and assign it to a new column in dataframe.
For simple example:
df = pd.DataFrame({k: np.random.randint(0, 1000, 100) for k in list('ABCDEFG')})
# drop duplicates
first = df.drop_duplicates(subset='A', keep='first').copy()
%timeit first['H'] = first['A']*first['B'] + first['C'] - first['D'].max()
this gives
532 µs ± 5.31 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
if I reset the index it becomes almost x2 faster (just in case the difference is due to some caching, I rerun with different order multiple times, it gave same result)
# drop duplicates but reset index
first = df.drop_duplicates(subset='A', keep='first').reset_index(drop=True).copy()
%timeit first['H'] = first['A']*first['B'] + first['C']
342 µs ± 7.47 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
Although it's not that bag difference, I wonder what causes this. Thanks.
UPDATE:
I redid this simple test, the issue was not index related, it seems like have something to do with the copy of a dataframe:
In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: df = pd.DataFrame({k: np.random.randint(0, 1000, 100) for k in list('ABCDEFG')})
In [4]: # drop duplicates
...: first = df.drop_duplicates(subset='A', keep='first').copy()
...: %timeit first['H'] = first['A']*first['B'] + first['C'] - first['D'].max()
558 µs ± 11.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [5]: # drop duplicates
...: first = df.drop_duplicates(subset='A', keep='first')
...: %timeit first['H'] = first['A']*first['B'] + first['C'] - first['D'].max()
/Users/sam/anaconda3/bin/ipython:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
#!/Users/sam_dessa/anaconda3/bin/python
20.7 ms ± 826 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
making a copy and assign a new column took ~ 532 µs but directly operate on the dataframe itself (which pandas also gave warning) gave 20.7 ms, same original question, what is causing this? Is it simply because the time spent on throwing out the warning?