Usually I create a new column df['c'] = df['a'] * df['b'] to calculate a production between column a and column b where df is a pandas's dataframe dtypes are float64. And pandas officially recommended this method instead of ".mul()" method. But when I run the code in the following I find bugs.
def func(sym):
location="D:\\data\\"
df=pd.read_csv(location+sym+".csv")
df['c']=df['A']*df['B'] # bug existing method (1)
# df['c']=df['A'].mul(df['B'],axis=0) #replacing method (2)
.....
for sym in symbollist:
func(sym)
I use code below to clean stocks data, obiviously df may be huge.But len(symbollist) only equals to 50. After codes have been run many times, method(1) may lead to random symbol's column c's value assigning to zero while method (2) perform well from the beginning to the end.
I use eclipse and Anaconda's newest version and python is 2.7. Pandas: 0.17.1, Numpy:1.10.1.
Related
I just wanted to know what is the difference in the function performed by these 2.
Data:
import pandas as pd
df = pd.DataFrame({"ID":["A","B","A","C","A","A","C","B"], "value":[1,2,4,3,6,7,3,4]})
as_index=False :
df_group1 = df.groupby("ID").sum().reset_index()
reset_index() :
df_group2 = df.groupby("ID", as_index=False).sum()
Both of them give the exact same output.
ID value
0 A 18
1 B 6
2 C 6
Can anyone tell me what is the difference and any example illustrating the same?
When you use as_index=False, you indicate to groupby() that you don't want to set the column ID as the index (duh!). When both implementation yield the same results, use as_index=False because it will save you some typing and an unnecessary pandas operation ;)
However, sometimes, you want to apply more complicated operations on your groups. In those occasions, you might find out that one is more suited than the other.
Example 1: You want to sum the values of three variables (i.e. columns) in a group on both axes.
Using as_index=True allows you to apply a sum over axis=1 without specifying the names of the columns, then summing the value over axis 0. When the operation is finished, you can use reset_index(drop=True/False) to get the dataframe under the right form.
Example 2: You need to set a value for the group based on the columns in the groupby().
Setting as_index=False allow you to check the condition on a common column and not on an index, which is often way easier.
At some point, you might come across KeyError when applying operations on groups. In that case, it is often because you are trying to use a column in your aggregate function that is currently an index of your GroupBy object.
I've noticed three methods of selecting a column in a Pandas DataFrame:
First method of selecting a column using loc:
df_new = df.loc[:, 'col1']
Second method - seems simpler and faster:
df_new = df['col1']
Third method - most convenient:
df_new = df.col1
Is there a difference between these three methods? I don't think so, in which case I'd rather use the third method.
I'm mostly curious as to why there appear to be three methods for doing the same thing.
In the following situations, they behave the same:
Selecting a single column (df['A'] is the same as df.loc[:, 'A'] -> selects column A)
Selecting a list of columns (df[['A', 'B', 'C']] is the same as df.loc[:, ['A', 'B', 'C']] -> selects columns A, B and C)
Slicing by rows (df[1:3] is the same as df.iloc[1:3] -> selects rows 1 and 2. Note, however, if you slice rows with loc, instead of iloc, you'll get rows 1, 2 and 3 assuming you have a RangeIndex. See details here.)
However, [] does not work in the following situations:
You can select a single row with df.loc[row_label]
You can select a list of rows with df.loc[[row_label1, row_label2]]
You can slice columns with df.loc[:, 'A':'C']
These three cannot be done with [].
More importantly, if your selection involves both rows and columns, then assignment becomes problematic.
df[1:3]['A'] = 5
This selects rows 1 and 2 then selects column 'A' of the returning object and assigns value 5 to it. The problem is, the returning object might be a copy so this may not change the actual DataFrame. This raises SettingWithCopyWarning. The correct way of making this assignment is:
df.loc[1:3, 'A'] = 5
With .loc, you are guaranteed to modify the original DataFrame. It also allows you to slice columns (df.loc[:, 'C':'F']), select a single row (df.loc[5]), and select a list of rows (df.loc[[1, 2, 5]]).
Also note that these two were not included in the API at the same time. .loc was added much later as a more powerful and explicit indexer. See unutbu's answer for more detail.
Note: Getting columns with [] vs . is a completely different topic. . is only there for convenience. It only allows accessing columns whose names are valid Python identifiers (i.e. they cannot contain spaces, they cannot be composed of numbers...). It cannot be used when the names conflict with Series/DataFrame methods. It also cannot be used for non-existing columns (i.e. the assignment df.a = 1 won't work if there is no column a). Other than that, . and [] are the same.
loc is specially useful when the index is not numeric (e.g. a DatetimeIndex) because you can get rows with particular labels from the index:
df.loc['2010-05-04 07:00:00']
df.loc['2010-1-1 0:00:00':'2010-12-31 23:59:59 ','Price']
However [] is intended to get columns with particular names:
df['Price']
With [] you can also filter rows, but it is more elaborated:
df[df['Date'] < datetime.datetime(2010,1,1,7,0,0)]['Price']
If you're confused which of these approaches is (at least) the recommended one for your use-case, take a look at this brief instructions from pandas tutorial:
When selecting subsets of data, square brackets [] are used.
Inside these brackets, you can use a single column/row label, a list
of column/row labels, a slice of labels, a conditional expression or
a colon.
Select specific rows and/or columns using loc when using the row and
column names
Select specific rows and/or columns using iloc when using the
positions in the table
You can assign new values to a selection based on loc/iloc.
I highlighted some of the points to make their use-case differences even more clear.
There seems to be a difference between df.loc[] and df[] when you create dataframe with multiple columns.
You can refer to this question:
Is there a nice way to generate multiple columns using .loc?
Here, you can't generate multiple columns using df.loc[:,['name1','name2']] but you can do by just using double bracket df[['name1','name2']]. (I wonder why they behave differently.)
I've noticed three methods of selecting a column in a Pandas DataFrame:
First method of selecting a column using loc:
df_new = df.loc[:, 'col1']
Second method - seems simpler and faster:
df_new = df['col1']
Third method - most convenient:
df_new = df.col1
Is there a difference between these three methods? I don't think so, in which case I'd rather use the third method.
I'm mostly curious as to why there appear to be three methods for doing the same thing.
In the following situations, they behave the same:
Selecting a single column (df['A'] is the same as df.loc[:, 'A'] -> selects column A)
Selecting a list of columns (df[['A', 'B', 'C']] is the same as df.loc[:, ['A', 'B', 'C']] -> selects columns A, B and C)
Slicing by rows (df[1:3] is the same as df.iloc[1:3] -> selects rows 1 and 2. Note, however, if you slice rows with loc, instead of iloc, you'll get rows 1, 2 and 3 assuming you have a RangeIndex. See details here.)
However, [] does not work in the following situations:
You can select a single row with df.loc[row_label]
You can select a list of rows with df.loc[[row_label1, row_label2]]
You can slice columns with df.loc[:, 'A':'C']
These three cannot be done with [].
More importantly, if your selection involves both rows and columns, then assignment becomes problematic.
df[1:3]['A'] = 5
This selects rows 1 and 2 then selects column 'A' of the returning object and assigns value 5 to it. The problem is, the returning object might be a copy so this may not change the actual DataFrame. This raises SettingWithCopyWarning. The correct way of making this assignment is:
df.loc[1:3, 'A'] = 5
With .loc, you are guaranteed to modify the original DataFrame. It also allows you to slice columns (df.loc[:, 'C':'F']), select a single row (df.loc[5]), and select a list of rows (df.loc[[1, 2, 5]]).
Also note that these two were not included in the API at the same time. .loc was added much later as a more powerful and explicit indexer. See unutbu's answer for more detail.
Note: Getting columns with [] vs . is a completely different topic. . is only there for convenience. It only allows accessing columns whose names are valid Python identifiers (i.e. they cannot contain spaces, they cannot be composed of numbers...). It cannot be used when the names conflict with Series/DataFrame methods. It also cannot be used for non-existing columns (i.e. the assignment df.a = 1 won't work if there is no column a). Other than that, . and [] are the same.
loc is specially useful when the index is not numeric (e.g. a DatetimeIndex) because you can get rows with particular labels from the index:
df.loc['2010-05-04 07:00:00']
df.loc['2010-1-1 0:00:00':'2010-12-31 23:59:59 ','Price']
However [] is intended to get columns with particular names:
df['Price']
With [] you can also filter rows, but it is more elaborated:
df[df['Date'] < datetime.datetime(2010,1,1,7,0,0)]['Price']
If you're confused which of these approaches is (at least) the recommended one for your use-case, take a look at this brief instructions from pandas tutorial:
When selecting subsets of data, square brackets [] are used.
Inside these brackets, you can use a single column/row label, a list
of column/row labels, a slice of labels, a conditional expression or
a colon.
Select specific rows and/or columns using loc when using the row and
column names
Select specific rows and/or columns using iloc when using the
positions in the table
You can assign new values to a selection based on loc/iloc.
I highlighted some of the points to make their use-case differences even more clear.
There seems to be a difference between df.loc[] and df[] when you create dataframe with multiple columns.
You can refer to this question:
Is there a nice way to generate multiple columns using .loc?
Here, you can't generate multiple columns using df.loc[:,['name1','name2']] but you can do by just using double bracket df[['name1','name2']]. (I wonder why they behave differently.)
Consider this case:
Python pandas equvilant to R groupby mutate
In dplyr:
df = df%>% group_by(a,b) %>%
means first the dataframe is grouped by column a then by b.
In my case I am trying to group my data first by group_name column, then by user_name , then by type_of_work . There are more than three columns (which is why I got confused) but I need data grouped according to these three headers in the same order. I already have an algorithm to work with columns after this stage. I only need an algorithm for creating a dataframe grouped according to these three columns.
It is important in my case that the sequence is preserved like the dplyr function.
Do we have anything similar in pandas data-frame?
Grouped = df.groupby(['a', 'b'])
Read more on "split-apply-combine" strategy in the pandas docs to see how pandas deals with these issues compared to R.
From your comment it seem you want assign the grouped frames. You can either use a groupbyobject through the API, eg grouped.mean(), or you can iterate through the groupby object. You will get name and group in each loop.
Here is my Pandas DataFrame:
import pandas as pd
dfa = df = pd.read_csv("twitDB3__org.csv")
dfa.drop([7-100], axis=0, inplace=True)
Output
ValueError: labels [-93] not contained in axis
I am new to canopy and want to delete a range of rows and it seems to require each row individual. Would appreciate any help
a) I think you want dfa.drop(range(7,101),... (What you did was just subtract 100 from 7 and pass the result (-93) as the label to drop.)
b) Note that this will also change df, because as you've written it, df and dfa are just two names for the same mutable object. If you want to end up with two different dataframes, then either make an explicit copy, or don't use inplace, and save the result: df2 = df.drop(...
c) This is a pandas question, not a canopy question. Canopy provides 500+ Python packages, and while it's true that pandas is one of the more popular of these, there is a whole pandas community out there.