I want to get a list of the column headers from a Pandas DataFrame. The DataFrame will come from user input, so I won't know how many columns there will be or what they will be called.
For example, if I'm given a DataFrame like this:
>>> my_dataframe
y gdp cap
0 1 2 5
1 2 3 9
2 8 7 2
3 3 4 7
4 6 7 7
5 4 8 3
6 8 2 8
7 9 9 10
8 6 6 4
9 10 10 7
I would get a list like this:
>>> header_list
['y', 'gdp', 'cap']
You can get the values as a list by doing:
list(my_dataframe.columns.values)
Also you can simply use (as shown in Ed Chum's answer):
list(my_dataframe)
There is a built-in method which is the most performant:
my_dataframe.columns.values.tolist()
.columns returns an Index, .columns.values returns an array and this has a helper function .tolist to return a list.
If performance is not as important to you, Index objects define a .tolist() method that you can call directly:
my_dataframe.columns.tolist()
The difference in performance is obvious:
%timeit df.columns.tolist()
16.7 µs ± 317 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit df.columns.values.tolist()
1.24 µs ± 12.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
For those who hate typing, you can just call list on df, as so:
list(df)
I did some quick tests, and perhaps unsurprisingly the built-in version using dataframe.columns.values.tolist() is the fastest:
In [1]: %timeit [column for column in df]
1000 loops, best of 3: 81.6 µs per loop
In [2]: %timeit df.columns.values.tolist()
10000 loops, best of 3: 16.1 µs per loop
In [3]: %timeit list(df)
10000 loops, best of 3: 44.9 µs per loop
In [4]: % timeit list(df.columns.values)
10000 loops, best of 3: 38.4 µs per loop
(I still really like the list(dataframe) though, so thanks EdChum!)
It gets even simpler (by Pandas 0.16.0):
df.columns.tolist()
will give you the column names in a nice list.
Extended Iterable Unpacking (Python 3.5+): [*df] and Friends
Unpacking generalizations (PEP 448) have been introduced with Python 3.5. So, the following operations are all possible.
df = pd.DataFrame('x', columns=['A', 'B', 'C'], index=range(5))
df
A B C
0 x x x
1 x x x
2 x x x
3 x x x
4 x x x
If you want a list....
[*df]
# ['A', 'B', 'C']
Or, if you want a set,
{*df}
# {'A', 'B', 'C'}
Or, if you want a tuple,
*df, # Please note the trailing comma
# ('A', 'B', 'C')
Or, if you want to store the result somewhere,
*cols, = df # A wild comma appears, again
cols
# ['A', 'B', 'C']
... if you're the kind of person who converts coffee to typing sounds, well, this is going consume your coffee more efficiently ;)
P.S.: if performance is important, you will want to ditch the
solutions above in favour of
df.columns.to_numpy().tolist()
# ['A', 'B', 'C']
This is similar to Ed Chum's answer, but updated for
v0.24 where .to_numpy() is preferred to the use of .values. See
this answer (by me) for more information.
Visual Check
Since I've seen this discussed in other answers, you can use iterable unpacking (no need for explicit loops).
print(*df)
A B C
print(*df, sep='\n')
A
B
C
Critique of Other Methods
Don't use an explicit for loop for an operation that can be done in a single line (list comprehensions are okay).
Next, using sorted(df) does not preserve the original order of the columns. For that, you should use list(df) instead.
Next, list(df.columns) and list(df.columns.values) are poor suggestions (as of the current version, v0.24). Both Index (returned from df.columns) and NumPy arrays (returned by df.columns.values) define .tolist() method which is faster and more idiomatic.
Lastly, listification i.e., list(df) should only be used as a concise alternative to the aforementioned methods for Python 3.4 or earlier where extended unpacking is not available.
>>> list(my_dataframe)
['y', 'gdp', 'cap']
To list the columns of a dataframe while in debugger mode, use a list comprehension:
>>> [c for c in my_dataframe]
['y', 'gdp', 'cap']
By the way, you can get a sorted list simply by using sorted:
>>> sorted(my_dataframe)
['cap', 'gdp', 'y']
That's available as my_dataframe.columns.
It's interesting, but df.columns.values.tolist() is almost three times faster than df.columns.tolist(), but I thought that they were the same:
In [97]: %timeit df.columns.values.tolist()
100000 loops, best of 3: 2.97 µs per loop
In [98]: %timeit df.columns.tolist()
10000 loops, best of 3: 9.67 µs per loop
A DataFrame follows the dict-like convention of iterating over the “keys” of the objects.
my_dataframe.keys()
Create a list of keys/columns - object method to_list() and the Pythonic way:
my_dataframe.keys().to_list()
list(my_dataframe.keys())
Basic iteration on a DataFrame returns column labels:
[column for column in my_dataframe]
Do not convert a DataFrame into a list, just to get the column labels. Do not stop thinking while looking for convenient code samples.
xlarge = pd.DataFrame(np.arange(100000000).reshape(10000,10000))
list(xlarge) # Compute time and memory consumption depend on dataframe size - O(N)
list(xlarge.keys()) # Constant time operation - O(1)
In the Notebook
For data exploration in the IPython notebook, my preferred way is this:
sorted(df)
Which will produce an easy to read alphabetically ordered list.
In a code repository
In code I find it more explicit to do
df.columns
Because it tells others reading your code what you are doing.
%%timeit
final_df.columns.values.tolist()
948 ns ± 19.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
%%timeit
list(final_df.columns)
14.2 µs ± 79.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%%timeit
list(final_df.columns.values)
1.88 µs ± 11.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
%%timeit
final_df.columns.tolist()
12.3 µs ± 27.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%%timeit
list(final_df.head(1).columns)
163 µs ± 20.6 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
The simplest option would be:
list(my_dataframe.columns) or my_dataframe.columns.tolist()
No need for the complex stuff above :)
Its very simple.
Like you can do it as:
list(df.columns)
For a quick, neat, visual check, try this:
for col in df.columns:
print col
As answered by Simeon Visser, you could do
list(my_dataframe.columns.values)
or
list(my_dataframe) # For less typing.
But I think most the sweet spot is:
list(my_dataframe.columns)
It is explicit and at the same time not unnecessarily long.
I feel the question deserves an additional explanation.
As fixxxer noted, the answer depends on the Pandas version you are using in your project. Which you can get with pd.__version__ command.
If you are for some reason like me (on Debian 8 (Jessie) I use 0.14.1) using an older version of Pandas than 0.16.0, then you need to use:
df.keys().tolist() because there isn’t any df.columns method implemented yet.
The advantage of this keys method is that it works even in newer version of Pandas, so it's more universal.
import pandas as pd
# create test dataframe
df = pd.DataFrame('x', columns=['A', 'B', 'C'], index=range(2))
list(df.columns)
Returns
['A', 'B', 'C']
n = []
for i in my_dataframe.columns:
n.append(i)
print n
This is the easiest way to reach your goal.
my_dataframe.columns.values.tolist()
and if you are Lazy, try this >
list(my_dataframe)
If the DataFrame happens to have an Index or MultiIndex and you want those included as column names too:
names = list(filter(None, df.index.names + df.columns.values.tolist()))
It avoids calling reset_index() which has an unnecessary performance hit for such a simple operation.
I've run into needing this more often because I'm shuttling data from databases where the dataframe index maps to a primary/unique key, but is really just another "column" to me. It would probably make sense for pandas to have a built-in method for something like this (totally possible I've missed it).
its the simple code for you :
for i in my_dataframe:
print(i)
just do it
Even though the solution that was provided previously is nice, I would also expect something like frame.column_names() to be a function in Pandas, but since it is not, maybe it would be nice to use the following syntax. It somehow preserves the feeling that you are using pandas in a proper way by calling the "tolist" function: frame.columns.tolist()
frame.columns.tolist()
listHeaders = [colName for colName in my_dataframe]
Related
I have the following DataFrame:
import pandas as pd
df = pd.DataFrame({'name': ["John", "Jack", "Jeff", "Kate"], "hobbies":[["pirates"], ["pirates"], ["climbing", "yoga"], ["yoga"]]})
# name hobbies
# 0 John [pirates]
# 1 Jack [pirates]
# 2 Jeff [climbing, yoga]
# 3 Kate [yoga]
I would like to have a list of the unique lists in hobbies.
Just to be clear, I don't want the list of unique hobbies (i.e. ["pirates", "climbing", "yoga"]), which is already covered in several questions including this one: pandas get unique values from column of lists
I would like instead the list [['pirates'], ['yoga'], ['climbing', 'yoga']].
I have thought of the following way but that does not seem very "panda-ic":
[list(t) for t in {tuple(h) for h in df["hobbies"]}]
Is there a better way to do it?
Let us change the list to tuple so we can do drop_duplicates
out = df.hobbies.apply(tuple).drop_duplicates().apply(list).tolist()
Out[143]: [['pirates'], ['climbing', 'yoga'], ['yoga']]
If you do not need converting back to list, you could do:
df.hobbies.apply(tuple).unique()
You could use numpy to do it:
import numpy as np
np.unique(df['hobbies'].to_numpy()).tolist()
lists aren't hashable keys, use a tuple instead and then convert to list
[*map(list,df['hobbies'].map(tuple).unique())]
output:
[['pirates'], ['climbing', 'yoga'], ['yoga']]
the use of unpacking over calling a list on a map object has proven faster for me
%%timeit
list(map(list,df['hobbies'].map(tuple).unique()))
385 µs ± 67.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%%timeit
[*map(list,df['hobbies'].map(tuple).unique())]
296 µs ± 15.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
I have seen many answers posted to questions on Stack Overflow involving the use of the Pandas method apply. I have also seen users commenting under them saying that "apply is slow, and should be avoided".
I have read many articles on the topic of performance that explain apply is slow. I have also seen a disclaimer in the docs about how apply is simply a convenience function for passing UDFs (can't seem to find that now). So, the general consensus is that apply should be avoided if possible. However, this raises the following questions:
If apply is so bad, then why is it in the API?
How and when should I make my code apply-free?
Are there ever any situations where apply is good (better than other possible solutions)?
apply, the Convenience Function you Never Needed
We start by addressing the questions in the OP, one by one.
"If apply is so bad, then why is it in the API?"
DataFrame.apply and Series.apply are convenience functions defined on DataFrame and Series object respectively. apply accepts any user defined function that applies a transformation/aggregation on a DataFrame. apply is effectively a silver bullet that does whatever any existing pandas function cannot do.
Some of the things apply can do:
Run any user-defined function on a DataFrame or Series
Apply a function either row-wise (axis=1) or column-wise (axis=0) on a DataFrame
Perform index alignment while applying the function
Perform aggregation with user-defined functions (however, we usually prefer agg or transform in these cases)
Perform element-wise transformations
Broadcast aggregated results to original rows (see the result_type argument).
Accept positional/keyword arguments to pass to the user-defined functions.
...Among others. For more information, see Row or Column-wise Function Application in the documentation.
So, with all these features, why is apply bad? It is because apply is slow. Pandas makes no assumptions about the nature of your function, and so iteratively applies your function to each row/column as necessary. Additionally, handling all of the situations above means apply incurs some major overhead at each iteration. Further, apply consumes a lot more memory, which is a challenge for memory bounded applications.
There are very few situations where apply is appropriate to use (more on that below). If you're not sure whether you should be using apply, you probably shouldn't.
Let's address the next question.
"How and when should I make my code apply-free?"
To rephrase, here are some common situations where you will want to get rid of any calls to apply.
Numeric Data
If you're working with numeric data, there is likely already a vectorized cython function that does exactly what you're trying to do (if not, please either ask a question on Stack Overflow or open a feature request on GitHub).
Contrast the performance of apply for a simple addition operation.
df = pd.DataFrame({"A": [9, 4, 2, 1], "B": [12, 7, 5, 4]})
df
A B
0 9 12
1 4 7
2 2 5
3 1 4
<!- ->
df.apply(np.sum)
A 16
B 28
dtype: int64
df.sum()
A 16
B 28
dtype: int64
Performance wise, there's no comparison, the cythonized equivalent is much faster. There's no need for a graph, because the difference is obvious even for toy data.
%timeit df.apply(np.sum)
%timeit df.sum()
2.22 ms ± 41.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
471 µs ± 8.16 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
Even if you enable passing raw arrays with the raw argument, it's still twice as slow.
%timeit df.apply(np.sum, raw=True)
840 µs ± 691 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Another example:
df.apply(lambda x: x.max() - x.min())
A 8
B 8
dtype: int64
df.max() - df.min()
A 8
B 8
dtype: int64
%timeit df.apply(lambda x: x.max() - x.min())
%timeit df.max() - df.min()
2.43 ms ± 450 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
1.23 ms ± 14.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In general, seek out vectorized alternatives if possible.
String/Regex
Pandas provides "vectorized" string functions in most situations, but there are rare cases where those functions do not... "apply", so to speak.
A common problem is to check whether a value in a column is present in another column of the same row.
df = pd.DataFrame({
'Name': ['mickey', 'donald', 'minnie'],
'Title': ['wonderland', "welcome to donald's castle", 'Minnie mouse clubhouse'],
'Value': [20, 10, 86]})
df
Name Value Title
0 mickey 20 wonderland
1 donald 10 welcome to donald's castle
2 minnie 86 Minnie mouse clubhouse
This should return the row second and third row, since "donald" and "minnie" are present in their respective "Title" columns.
Using apply, this would be done using
df.apply(lambda x: x['Name'].lower() in x['Title'].lower(), axis=1)
0 False
1 True
2 True
dtype: bool
df[df.apply(lambda x: x['Name'].lower() in x['Title'].lower(), axis=1)]
Name Title Value
1 donald welcome to donald's castle 10
2 minnie Minnie mouse clubhouse 86
However, a better solution exists using list comprehensions.
df[[y.lower() in x.lower() for x, y in zip(df['Title'], df['Name'])]]
Name Title Value
1 donald welcome to donald's castle 10
2 minnie Minnie mouse clubhouse 86
<!- ->
%timeit df[df.apply(lambda x: x['Name'].lower() in x['Title'].lower(), axis=1)]
%timeit df[[y.lower() in x.lower() for x, y in zip(df['Title'], df['Name'])]]
2.85 ms ± 38.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
788 µs ± 16.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
The thing to note here is that iterative routines happen to be faster than apply, because of the lower overhead. If you need to handle NaNs and invalid dtypes, you can build on this using a custom function you can then call with arguments inside the list comprehension.
For more information on when list comprehensions should be considered a good option, see my writeup: Are for-loops in pandas really bad? When should I care?.
Note
Date and datetime operations also have vectorized versions. So, for example, you should prefer pd.to_datetime(df['date']), over,
say, df['date'].apply(pd.to_datetime).
Read more at the
docs.
A Common Pitfall: Exploding Columns of Lists
s = pd.Series([[1, 2]] * 3)
s
0 [1, 2]
1 [1, 2]
2 [1, 2]
dtype: object
People are tempted to use apply(pd.Series). This is horrible in terms of performance.
s.apply(pd.Series)
0 1
0 1 2
1 1 2
2 1 2
A better option is to listify the column and pass it to pd.DataFrame.
pd.DataFrame(s.tolist())
0 1
0 1 2
1 1 2
2 1 2
<!- ->
%timeit s.apply(pd.Series)
%timeit pd.DataFrame(s.tolist())
2.65 ms ± 294 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
816 µs ± 40.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
Lastly,
"Are there any situations where apply is good?"
Apply is a convenience function, so there are situations where the overhead is negligible enough to forgive. It really depends on how many times the function is called.
Functions that are Vectorized for Series, but not DataFrames
What if you want to apply a string operation on multiple columns? What if you want to convert multiple columns to datetime? These functions are vectorized for Series only, so they must be applied over each column that you want to convert/operate on.
df = pd.DataFrame(
pd.date_range('2018-12-31','2019-01-31', freq='2D').date.astype(str).reshape(-1, 2),
columns=['date1', 'date2'])
df
date1 date2
0 2018-12-31 2019-01-02
1 2019-01-04 2019-01-06
2 2019-01-08 2019-01-10
3 2019-01-12 2019-01-14
4 2019-01-16 2019-01-18
5 2019-01-20 2019-01-22
6 2019-01-24 2019-01-26
7 2019-01-28 2019-01-30
df.dtypes
date1 object
date2 object
dtype: object
This is an admissible case for apply:
df.apply(pd.to_datetime, errors='coerce').dtypes
date1 datetime64[ns]
date2 datetime64[ns]
dtype: object
Note that it would also make sense to stack, or just use an explicit loop. All these options are slightly faster than using apply, but the difference is small enough to forgive.
%timeit df.apply(pd.to_datetime, errors='coerce')
%timeit pd.to_datetime(df.stack(), errors='coerce').unstack()
%timeit pd.concat([pd.to_datetime(df[c], errors='coerce') for c in df], axis=1)
%timeit for c in df.columns: df[c] = pd.to_datetime(df[c], errors='coerce')
5.49 ms ± 247 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
3.94 ms ± 48.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
3.16 ms ± 216 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
2.41 ms ± 1.71 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
You can make a similar case for other operations such as string operations, or conversion to category.
u = df.apply(lambda x: x.str.contains(...))
v = df.apply(lambda x: x.astype(category))
v/s
u = pd.concat([df[c].str.contains(...) for c in df], axis=1)
v = df.copy()
for c in df:
v[c] = df[c].astype(category)
And so on...
Converting Series to str: astype versus apply
This seems like an idiosyncrasy of the API. Using apply to convert integers in a Series to string is comparable (and sometimes faster) than using astype.
The graph was plotted using the perfplot library.
import perfplot
perfplot.show(
setup=lambda n: pd.Series(np.random.randint(0, n, n)),
kernels=[
lambda s: s.astype(str),
lambda s: s.apply(str)
],
labels=['astype', 'apply'],
n_range=[2**k for k in range(1, 20)],
xlabel='N',
logx=True,
logy=True,
equality_check=lambda x, y: (x == y).all())
With floats, I see the astype is consistently as fast as, or slightly faster than apply. So this has to do with the fact that the data in the test is integer type.
GroupBy operations with chained transformations
GroupBy.apply has not been discussed until now, but GroupBy.apply is also an iterative convenience function to handle anything that the existing GroupBy functions do not.
One common requirement is to perform a GroupBy and then two prime operations such as a "lagged cumsum":
df = pd.DataFrame({"A": list('aabcccddee'), "B": [12, 7, 5, 4, 5, 4, 3, 2, 1, 10]})
df
A B
0 a 12
1 a 7
2 b 5
3 c 4
4 c 5
5 c 4
6 d 3
7 d 2
8 e 1
9 e 10
<!- ->
You'd need two successive groupby calls here:
df.groupby('A').B.cumsum().groupby(df.A).shift()
0 NaN
1 12.0
2 NaN
3 NaN
4 4.0
5 9.0
6 NaN
7 3.0
8 NaN
9 1.0
Name: B, dtype: float64
Using apply, you can shorten this to a a single call.
df.groupby('A').B.apply(lambda x: x.cumsum().shift())
0 NaN
1 12.0
2 NaN
3 NaN
4 4.0
5 9.0
6 NaN
7 3.0
8 NaN
9 1.0
Name: B, dtype: float64
It is very hard to quantify the performance because it depends on the data. But in general, apply is an acceptable solution if the goal is to reduce a groupby call (because groupby is also quite expensive).
Other Caveats
Aside from the caveats mentioned above, it is also worth mentioning that apply operates on the first row (or column) twice. This is done to determine whether the function has any side effects. If not, apply may be able to use a fast-path for evaluating the result, else it falls back to a slow implementation.
df = pd.DataFrame({
'A': [1, 2],
'B': ['x', 'y']
})
def func(x):
print(x['A'])
return x
df.apply(func, axis=1)
# 1
# 1
# 2
A B
0 1 x
1 2 y
This behaviour is also seen in GroupBy.apply on pandas versions <0.25 (it was fixed for 0.25, see here for more information.)
Not all applys are alike
The below chart suggests when to consider apply1. Green means possibly efficient; red avoid.
Some of this is intuitive: pd.Series.apply is a Python-level row-wise loop, ditto pd.DataFrame.apply row-wise (axis=1). The misuses of these are many and wide-ranging. The other post deals with them in more depth. Popular solutions are to use vectorised methods, list comprehensions (assumes clean data), or efficient tools such as the pd.DataFrame constructor (e.g. to avoid apply(pd.Series)).
If you are using pd.DataFrame.apply row-wise, specifying raw=True (where possible) is often beneficial. At this stage, numba is usually a better choice.
GroupBy.apply: generally favoured
Repeating groupby operations to avoid apply will hurt performance. GroupBy.apply is usually fine here, provided the methods you use in your custom function are themselves vectorised. Sometimes there is no native Pandas method for a groupwise aggregation you wish to apply. In this case, for a small number of groups apply with a custom function may still offer reasonable performance.
pd.DataFrame.apply column-wise: a mixed bag
pd.DataFrame.apply column-wise (axis=0) is an interesting case. For a small number of rows versus a large number of columns, it's almost always expensive. For a large number of rows relative to columns, the more common case, you may sometimes see significant performance improvements using apply:
# Python 3.7, Pandas 0.23.4
np.random.seed(0)
df = pd.DataFrame(np.random.random((10**7, 3))) # Scenario_1, many rows
df = pd.DataFrame(np.random.random((10**4, 10**3))) # Scenario_2, many columns
# Scenario_1 | Scenario_2
%timeit df.sum() # 800 ms | 109 ms
%timeit df.apply(pd.Series.sum) # 568 ms | 325 ms
%timeit df.max() - df.min() # 1.63 s | 314 ms
%timeit df.apply(lambda x: x.max() - x.min()) # 838 ms | 473 ms
%timeit df.mean() # 108 ms | 94.4 ms
%timeit df.apply(pd.Series.mean) # 276 ms | 233 ms
1 There are exceptions, but these are usually marginal or uncommon. A couple of examples:
df['col'].apply(str) may slightly outperform df['col'].astype(str).
df.apply(pd.to_datetime) working on strings doesn't scale well with rows versus a regular for loop.
For axis=1 (i.e. row-wise functions) then you can just use the following function in lieu of apply. I wonder why this isn't the pandas behavior. (Untested with compound indexes, but it does appear to be much faster than apply)
def faster_df_apply(df, func):
cols = list(df.columns)
data, index = [], []
for row in df.itertuples(index=True):
row_dict = {f:v for f,v in zip(cols, row[1:])}
data.append(func(row_dict))
index.append(row[0])
return pd.Series(data, index=index)
Are there ever any situations where apply is good?
Yes, sometimes.
Task: decode Unicode strings.
import numpy as np
import pandas as pd
import unidecode
s = pd.Series(['mañana','Ceñía'])
s.head()
0 mañana
1 Ceñía
s.apply(unidecode.unidecode)
0 manana
1 Cenia
Update
I was by no means advocating for the use of apply, just thinking since the NumPy cannot deal with the above situation, it could have been a good candidate for pandas apply. But I was forgetting the plain ol list comprehension thanks to the reminder by #jpp.
I am a bit surprised that for a unique dtype DataFrame (nxn dataFrame), it is slower to access a row than a column. From what I gather a DataFrame of identical dtype should be stored as a contiguous block in memory, so accessing rows or columns should be equally as fast (just a matter of updating the correct stride).
Sample code:
df = pd.DataFrame(np.random.randn(100, 100))
%timeit df[0]
%timeit df.loc[0]
The slowest run took 12.86 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 2.72 µs per loop
10000 loops, best of 3: 116 µs per loop
There is definitely something I dont understand well about how a dataFrame is stored, thanks for your help !
I'm not an expert in the implementation details of Pandas, but I've used it enough that I can make an educated guess.
As I understand it, the Pandas data structure is most directly comparable to a dictionary of dictionaries, where the first index is the columns. Thus, the DF:
a b
c 1 2
d 3 4
is essentially {'a': {'c': 1, 'd': 3}, 'b': {'c': 2, 'd': 4}}. I'll assume I'm correct about that assertion from here on out, and would love to be corrected if someone knows more about pandas.
Thus, indexing a column is a simple hash lookup, whereas indexing a row requires iterating over all columns and doing a hash lookup for each one.
I think the reasoning is that this makes it really efficient to access a particular attribute of all rows and add new columns, which is normally how you interact with a dataframe. For such tabular use cases, it's much faster than a simple matrix layout, since you don't have to stride through memory (a whole column is stored more or less locally), but of course that's a tradeoff that makes interacting with rows less efficient (hence why it's not as easy syntactically to do so; you'll note that most Pandas operations default to interacting with columns, and interacting with rows is more or less a secondary objective in the module).
If you look at the underlying numpy array, you'll see that access is the same speed for rows / columns, at least in my test:
%timeit df.values[0]
# 10.2 µs ± 596 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit df.values[:, 0]
# 10.2 µs ± 730 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
Series (columns) are more first-class citizens in a dataframe than rows are. I think accessing the columns is more like a dictionary lookup, which is why it's so fast. Usually there are few columns, and each is meaningful, so it makes sense to store them this way. There are often very many rows, though, and an individual row doesn't have as much significance. This is a bit of conjecture, though. You'd have to go look at the source code to see what is actually being called each time and determine from that why the operations take a different amount of time - maybe an answer will pop up with that later.
Here's another timing comparison:
%timeit df.iloc[0, :]
# 141 µs ± 7 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit df.iloc[:, 0]
# 61.9 µs ± 1.76 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
Accessing the columns is quicker this way too, though much slower. I'm not sure what would explain this. I assume that the slowdown compared with accessing a row/column directly comes from needing to return a pd.Series. When accessing a row, a new pd.Series might need to be created. But I don't know why iloc is slower for columns too - perhaps it also creates a new series each time, since iloc can be used quite flexibly and might not return an existing series (or could return a dataframe). But if a new series is created both times, then I'm again at a loss for why one operation beats the other.
And for more completeness
%timeit df.loc[0, :]
# 155 µs ± 6.48 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit df.loc[:, 0]
# 35.6 µs ± 1.28 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
I have a pandas dataframe A with column keywords as
(here Im showing only 4 rows but in actual there are millions) :-
keywords
['loans','mercedez','bugatti']
['trump','usa']
['galaxy','7s','canon','macbook']
['beiber','spiderman','marvels','ironmen']
I want to sum total number of list elements in column keywords and store it into some variable. Something like
total_sum=elements in keywords[0]+elements in keywords[1]+elements in
keywords[2]+elements in keywords[3]
total_sum=3+2+4+4
total_sum=13
How I can do it in pandas?
IIUC
Setup
df = pd.DataFrame()
df['keywords']=[['loans','mercedez','bugatti'],
['trump','usa'],
['galaxy','7s','canon','macbook'],
['beiber','spiderman','marvels','ironmen']]
Then juse use str.len and sum
df.keywords.str.len().sum()
Detail:
df.keywords.str.len()
0 3
1 2
2 4
3 4
Name: keywords, dtype: int64
Ps: If you have strings that look like a list, use ast.literal_eval to convert to list first.
df.keywords.transform(ast.literal_eval).str.len().sum()
Using sum and map:
sum(map(len, df.keywords))
Sample
df = pd.DataFrame({
'keywords': [['a', 'b', 'c'], ['c', 'd'], ['a', 'b', 'c', 'd'], ['g', 'h', 'i']]
})
sum(map(len, df.keywords))
12
Timings
df = pd.concat([df]*10000)
%timeit sum(map(len, df.keywords))
1.87 ms ± 52.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit df.keywords.map(len).sum()
13.5 ms ± 661 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%timeit df.keywords.str.len().sum()
14.3 ms ± 272 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Validation
>>> sum(map(len, df.keywords)) == df.keywords.map(len).sum() == df.keywords.str.len().sum()
True
A bit of a disclaimer: using pandas methods on columns that contain lists is always going to be inefficient (which is why using non-pandas' methods is so much faster here), since DataFrames are not meant to store list. You should try to avoid this whenever possible.
You can try this one:
df.keywords.map(len).sum()
Simple as that.
Maybe Pandas evolved since then.
df['len_of_list'] = df.my_columns_with_list.agg([len])
Cheers,
I want to sum total number of list elements in column keywords
This is different from what you pseudo-coded. I believe you mean to call the size function for dataframes:
total_sum = keywords.size
Method 1:
len([item for sublist in df.keywords for item in sublist]
Method 2:
df.keywords.apply(len).sum()
.
df = [{"item": "a", "item_price": [1,1.5,2]}, {"item": "b", "item_price": [0.5,0.75,1]}]
df = pd.DataFrame(df)
print(df)
print("Ans:",len([item for sublist in df.item_price for item in sublist]))
OUTPUT
df
item item_price
0 a [1, 1.5, 2]
1 b [0.5, 0.75, 1]
Ans:6
More like a list flatten problem
import itertools
len(list(itertools.chain(*df.keywords.values.tolist())))
Out[57]: 13
I would like to select from a pandas dataframe specific columns using column index.
In particular, I would like to select columns index by the column index generated by c(12:26,69:85,96:99,134:928,933:935,940:967) in R. I wonder how can I do that in Python?
I am thinking something like the following, but of course, python does not have a function called c()...
input2 = input2.iloc[:,c(12:26,69:85,96:99,134:928,933:935,940:967)]
The equivalent is numpy's r_. It combines integer slices without needing to call ranges for each of them:
np.r_[2:4, 7:11, 21:25]
Out: array([ 2, 3, 7, 8, 9, 10, 21, 22, 23, 24])
df = pd.DataFrame(np.random.randn(1000))
df.iloc[np.r_[2:4, 7:11, 21:25]]
Out:
0
2 2.720383
3 0.656391
7 -0.581855
8 0.047612
9 1.416250
10 0.206395
21 -1.519904
22 0.681153
23 -1.208401
24 -0.358545
Putting #hrbrmstr 's comment into an answer, because it solved my issue and I want to make it clear that this question is resolved. In addition, please note that range(a,b) gives the numbers (a, a+1, ..., b-2, b-1), and doesn't include b.
R's combine function
c(4,12:26,69:85,96:99,134:928,933:935)
is translated into Python as
[4] + list(range(12,27)) + list(range(69,86)) + list(range(96,100)) + list(range(134,929)) + list(range(933,936))
To answer the actual question,
Python equivalent of R c() function, for dataframe column indices?
I'm using this definition of c()
c = lambda v: v.split(',') if ":" not in v else eval(f'np.r_[{v}]')
Then we can do things like:
df = pd.DataFrame({'x': np.random.randn(1000),
'y': np.random.randn(1000)})
# row selection
df.iloc[c('2:4,7:11,21:25')]
# columns by name
df[c('x,y')]
# columns by range
df.T[c('12:15,17:25,500:750')]
That's pretty much as close as it gets in terms of R-like syntax.
To the curious mind
Note there is a performance penality in using c() as per above v.s. np.r_. To paraphrase Knuth, let's not optimize prematurely ;-)
%timeit np.r_[2:4, 7:11, 21:25]
27.3 µs ± 786 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit c("2:4, 7:11, 21:25")
53.7 µs ± 977 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)